howard.objects.variants API documentation

Variants( conn=None, input: str = None, output: str = None, config: dict = {}, param: dict = {}, load: bool = False) View Source

36    def __init__(
37        self,
38        conn=None,
39        input: str = None,
40        output: str = None,
41        config: dict = {},
42        param: dict = {},
43        load: bool = False,
44    ) -> None:
45        """
46        The function `__init__` initializes the variables, sets the input, output, config, param, connexion and
47        header
48
49        :param conn: the connection to the database
50        :param input: the input file
51        :param output: the output file
52        :param config: a dictionary containing the configuration of the model
53        :param param: a dictionary containing the parameters of the model
54        """
55
56        # Init variables
57        self.init_variables()
58
59        # Input
60        self.set_input(input)
61
62        # Config
63        self.set_config(config)
64
65        # Param
66        self.set_param(param)
67
68        # Output
69        self.set_output(output)
70
71        # connexion
72        self.set_connexion(conn)
73
74        # Header
75        self.set_header()
76
77        # Samples
78        self.set_samples()
79
80        # Load data
81        if load:
82            self.load_data()

The function __init__ initializes the variables, sets the input, output, config, param, connexion and header

Parameters

conn: the connection to the database
input: the input file
output: the output file
config: a dictionary containing the configuration of the model
param: a dictionary containing the parameters of the model

def set_samples(self, samples: list = None) -> list: View Source

 84    def set_samples(self, samples: list = None) -> list:
 85        """
 86        The function `set_samples` sets the samples attribute of an object to a provided list or
 87        retrieves it from a parameter dictionary.
 88
 89        :param samples: The `set_samples` method is a method of a class that takes a list of samples as
 90        input and sets the `samples` attribute of the class to the provided list. If no samples are
 91        provided, it tries to get the samples from the class's parameters using the `get_param` method
 92        :type samples: list
 93        :return: The `samples` list is being returned.
 94        """
 95
 96        if not samples:
 97            samples = self.get_param().get("samples", {}).get("list", None)
 98
 99        self.samples = samples
100
101        return samples

The function set_samples sets the samples attribute of an object to a provided list or retrieves it from a parameter dictionary.

Parameters

samples: The set_samples method is a method of a class that takes a list of samples as input and sets the samples attribute of the class to the provided list. If no samples are provided, it tries to get the samples from the class's parameters using the get_param method

Returns

The samples list is being returned.

def get_samples(self) -> list: View Source

103    def get_samples(self) -> list:
104        """
105        This function returns a list of samples.
106        :return: The `get_samples` method is returning the `samples` attribute of the object.
107        """
108
109        return self.samples

This function returns a list of samples.

Returns

The get_samples method is returning the samples attribute of the object.

def get_samples_check(self) -> bool: View Source

111    def get_samples_check(self) -> bool:
112        """
113        This function returns the value of the "check" key within the "samples" dictionary retrieved
114        from the parameters.
115        :return: The method `get_samples_check` is returning the value of the key "check" inside the
116        "samples" dictionary, which is nested inside the dictionary returned by the `get_param()`
117        method. If the key "check" is not found, it will return `False`.
118        """
119
120        return self.get_param().get("samples", {}).get("check", True)

This function returns the value of the "check" key within the "samples" dictionary retrieved from the parameters.

Returns

The method get_samples_check is returning the value of the key "check" inside the "samples" dictionary, which is nested inside the dictionary returned by the get_param() method. If the key "check" is not found, it will return False.

def set_input(self, input: str = None) -> None: View Source

122    def set_input(self, input: str = None) -> None:
123        """
124        The function `set_input` takes a file name as input, extracts the name and extension, and sets
125        attributes in the class accordingly.
126
127        :param input: The `set_input` method in the provided code snippet is used to set attributes
128        related to the input file. Here's a breakdown of the parameters and their usage in the method:
129        :type input: str
130        """
131
132        if input and not isinstance(input, str):
133            try:
134                self.input = input.name
135            except:
136                log.error(f"Input file '{input} in bad format")
137                raise ValueError(f"Input file '{input} in bad format")
138        else:
139            self.input = input
140
141        # Input format
142        if input:
143            input_name, input_extension = os.path.splitext(self.input)
144            self.input_name = input_name
145            self.input_extension = input_extension
146            self.input_format = self.input_extension.replace(".", "")

The function set_input takes a file name as input, extracts the name and extension, and sets attributes in the class accordingly.

Parameters

input: The set_input method in the provided code snippet is used to set attributes related to the input file. Here's a breakdown of the parameters and their usage in the method:

def set_config(self, config: dict) -> None: View Source

148    def set_config(self, config: dict) -> None:
149        """
150        The set_config function takes a config object and assigns it as the configuration object for the
151        class.
152
153        :param config: The `config` parameter in the `set_config` function is a dictionary object that
154        contains configuration settings for the class. When you call the `set_config` function with a
155        dictionary object as the argument, it will set that dictionary as the configuration object for
156        the class
157        :type config: dict
158        """
159
160        self.config = config

The set_config function takes a config object and assigns it as the configuration object for the class.

Parameters

config: The config parameter in the set_config function is a dictionary object that contains configuration settings for the class. When you call the set_config function with a dictionary object as the argument, it will set that dictionary as the configuration object for the class

def set_param(self, param: dict) -> None: View Source

162    def set_param(self, param: dict) -> None:
163        """
164        This function sets a parameter object for the class based on the input dictionary.
165
166        :param param: The `set_param` method you provided takes a dictionary object as input and sets it
167        as the `param` attribute of the class instance
168        :type param: dict
169        """
170
171        self.param = param

This function sets a parameter object for the class based on the input dictionary.

Parameters

param: The set_param method you provided takes a dictionary object as input and sets it as the param attribute of the class instance

def init_variables(self) -> None: View Source

173    def init_variables(self) -> None:
174        """
175        This function initializes the variables that will be used in the rest of the class
176        """
177
178        self.prefix = "howard"
179        self.table_variants = "variants"
180        self.dataframe = None
181
182        self.comparison_map = {
183            "gt": ">",
184            "gte": ">=",
185            "lt": "<",
186            "lte": "<=",
187            "equals": "=",
188            "contains": "SIMILAR TO",
189        }
190
191        self.code_type_map = {"Integer": 0, "String": 1, "Float": 2, "Flag": 3}
192
193        self.code_type_map_to_sql = {
194            "Integer": "INTEGER",
195            "String": "VARCHAR",
196            "Float": "FLOAT",
197            "Flag": "VARCHAR",
198        }
199
200        self.index_additionnal_fields = []

This function initializes the variables that will be used in the rest of the class

def get_indexing(self) -> bool: View Source

202    def get_indexing(self) -> bool:
203        """
204        It returns the value of the key "indexing" in the dictionary. If the key is not present, it
205        returns False.
206        :return: The value of the indexing parameter.
207        """
208
209        return self.get_param().get("indexing", False)

It returns the value of the key "indexing" in the dictionary. If the key is not present, it returns False.

Returns

The value of the indexing parameter.

def get_connexion_config(self) -> dict: View Source

211    def get_connexion_config(self) -> dict:
212        """
213        The function `get_connexion_config` returns a dictionary containing the configuration for a
214        connection, including the number of threads and memory limit.
215        :return: a dictionary containing the configuration for the Connexion library.
216        """
217
218        # config
219        config = self.get_config()
220
221        # Connexion config
222        connexion_config = {}
223        threads = self.get_threads()
224
225        # Threads
226        if threads:
227            connexion_config["threads"] = threads
228
229        # Memory
230        # if config.get("memory", None):
231        #     connexion_config["memory_limit"] = config.get("memory")
232        if self.get_memory():
233            connexion_config["memory_limit"] = self.get_memory()
234
235        # Temporary directory
236        if config.get("tmp", None):
237            connexion_config["temp_directory"] = config.get("tmp")
238
239        # Access
240        if config.get("access", None):
241            access = config.get("access")
242            if access in ["RO"]:
243                access = "READ_ONLY"
244            elif access in ["RW"]:
245                access = "READ_WRITE"
246            connexion_db = self.get_connexion_db()
247            if connexion_db in ":memory:":
248                access = "READ_WRITE"
249            connexion_config["access_mode"] = access
250
251        return connexion_config

The function get_connexion_config returns a dictionary containing the configuration for a connection, including the number of threads and memory limit.

Returns

a dictionary containing the configuration for the Connexion library.

def get_duckdb_settings(self) -> dict: View Source

253    def get_duckdb_settings(self) -> dict:
254        """
255        The function `get_duckdb_settings` retrieves DuckDB settings from a configuration file or a
256        string.
257        :return: The function `get_duckdb_settings` returns a dictionary object `duckdb_settings_dict`.
258        """
259
260        # config
261        config = self.get_config()
262
263        # duckdb settings
264        duckdb_settings_dict = {}
265        if config.get("duckdb_settings", None):
266            duckdb_settings = config.get("duckdb_settings")
267            duckdb_settings = full_path(duckdb_settings)
268            # duckdb setting is a file
269            if os.path.exists(duckdb_settings):
270                with open(duckdb_settings) as json_file:
271                    duckdb_settings_dict = yaml.safe_load(json_file)
272            # duckdb settings is a string
273            else:
274                duckdb_settings_dict = json.loads(duckdb_settings)
275
276        return duckdb_settings_dict

The function get_duckdb_settings retrieves DuckDB settings from a configuration file or a string.

Returns

The function get_duckdb_settings returns a dictionary object duckdb_settings_dict.

def set_connexion_db(self) -> str: View Source

278    def set_connexion_db(self) -> str:
279        """
280        The function `set_connexion_db` returns the appropriate database connection string based on the
281        input format and connection type.
282        :return: the value of the variable `connexion_db`.
283        """
284
285        # Default connexion db
286        default_connexion_db = ":memory:"
287
288        # Find connexion db
289        if self.get_input_format() in ["db", "duckdb"]:
290            connexion_db = self.get_input()
291        elif self.get_connexion_type() in ["memory", default_connexion_db, None]:
292            connexion_db = default_connexion_db
293        elif self.get_connexion_type() in ["tmpfile"]:
294            tmp_name = tempfile.mkdtemp(
295                prefix=self.get_prefix(), dir=self.get_tmp_dir(), suffix=".db"
296            )
297            connexion_db = f"{tmp_name}/tmp.db"
298        elif self.get_connexion_type() != "":
299            connexion_db = self.get_connexion_type()
300        else:
301            connexion_db = default_connexion_db
302
303        # Set connexion db
304        self.connexion_db = connexion_db
305
306        return connexion_db

The function set_connexion_db returns the appropriate database connection string based on the input format and connection type.

Returns

the value of the variable connexion_db.

def set_connexion(self, conn) -> None: View Source

308    def set_connexion(self, conn) -> None:
309        """
310        The function `set_connexion` creates a connection to a database, with options for different
311        database formats and settings.
312
313        :param conn: The `conn` parameter in the `set_connexion` method is the connection to the
314        database. If a connection is not provided, a new connection to an in-memory database is created.
315        The method then proceeds to set up the connection based on the specified format (e.g., duckdb or
316        sqlite
317        """
318
319        # Connexion db
320        connexion_db = self.set_connexion_db()
321
322        # Connexion config
323        connexion_config = self.get_connexion_config()
324
325        # Connexion format
326        connexion_format = self.get_config().get("connexion_format", "duckdb")
327        # Set connexion format
328        self.connexion_format = connexion_format
329
330        # Connexion
331        if not conn:
332            if connexion_format in ["duckdb"]:
333                conn = duckdb.connect(connexion_db, config=connexion_config)
334                # duckDB settings
335                duckdb_settings = self.get_duckdb_settings()
336                if duckdb_settings:
337                    for setting in duckdb_settings:
338                        setting_value = duckdb_settings.get(setting)
339                        if isinstance(setting_value, str):
340                            setting_value = f"'{setting_value}'"
341                        conn.execute(f"PRAGMA {setting}={setting_value};")
342            elif connexion_format in ["sqlite"]:
343                conn = sqlite3.connect(connexion_db)
344
345        # Set connexion
346        self.conn = conn
347
348        # Log
349        log.debug(f"connexion_format: {connexion_format}")
350        log.debug(f"connexion_db: {connexion_db}")
351        log.debug(f"connexion config: {connexion_config}")
352        log.debug(f"connexion duckdb settings: {self.get_duckdb_settings()}")

The function set_connexion creates a connection to a database, with options for different database formats and settings.

Parameters

conn: The conn parameter in the set_connexion method is the connection to the database. If a connection is not provided, a new connection to an in-memory database is created. The method then proceeds to set up the connection based on the specified format (e.g., duckdb or sqlite

def set_output(self, output: str = None) -> None: View Source

354    def set_output(self, output: str = None) -> None:
355        """
356        The `set_output` function in Python sets the output file based on the input or a specified key
357        in the config file, extracting the output name, extension, and format.
358
359        :param output: The `output` parameter in the `set_output` method is used to specify the name of
360        the output file. If the config file has an 'output' key, the method sets the output to the value
361        of that key. If no output is provided, it sets the output to `None`
362        :type output: str
363        """
364
365        if output and not isinstance(output, str):
366            self.output = output.name
367        else:
368            self.output = output
369
370        # Output format
371        if self.output:
372            output_name, output_extension = os.path.splitext(self.output)
373            self.output_name = output_name
374            self.output_extension = output_extension
375            self.output_format = self.output_extension.replace(".", "")
376        else:
377            self.output_name = None
378            self.output_extension = None
379            self.output_format = None

The set_output function in Python sets the output file based on the input or a specified key in the config file, extracting the output name, extension, and format.

Parameters

output: The output parameter in the set_output method is used to specify the name of the output file. If the config file has an 'output' key, the method sets the output to the value of that key. If no output is provided, it sets the output to None

def set_header(self) -> None: View Source

381    def set_header(self) -> None:
382        """
383        It reads the header of a VCF file and stores it as a list of strings and as a VCF object
384        """
385
386        input_file = self.get_input()
387        default_header_list = [
388            "##fileformat=VCFv4.2",
389            "#CHROM	POS	ID	REF	ALT	QUAL	FILTER	INFO",
390        ]
391
392        # Full path
393        input_file = full_path(input_file)
394
395        if input_file:
396
397            input_format = self.get_input_format()
398            input_compressed = self.get_input_compressed()
399            config = self.get_config()
400            header_list = default_header_list
401            if input_format in [
402                "vcf",
403                "hdr",
404                "tsv",
405                "csv",
406                "psv",
407                "parquet",
408                "db",
409                "duckdb",
410            ]:
411                # header provided in param
412                if config.get("header_file", None):
413                    with open(config.get("header_file"), "rt") as f:
414                        header_list = self.read_vcf_header(f)
415                # within a vcf file format (header within input file itsself)
416                elif input_format in ["vcf", "hdr"] and not os.path.isdir(input_file):
417                    # within a compressed vcf file format (.vcf.gz)
418                    if input_compressed:
419                        with bgzf.open(input_file, "rt") as f:
420                            header_list = self.read_vcf_header(f)
421                    # within an uncompressed vcf file format (.vcf)
422                    else:
423                        with open(input_file, "rt") as f:
424                            header_list = self.read_vcf_header(f)
425                # header provided in default external file .hdr
426                elif os.path.exists((input_file + ".hdr")):
427                    with open(input_file + ".hdr", "rt") as f:
428                        header_list = self.read_vcf_header(f)
429                else:
430                    try:  # Try to get header info fields and file columns
431
432                        with tempfile.TemporaryDirectory() as tmpdir:
433
434                            # Create database
435                            db_for_header = Database(database=input_file)
436
437                            # Get header columns for infos fields
438                            db_header_from_columns = (
439                                db_for_header.get_header_from_columns()
440                            )
441
442                            # Get real columns in the file
443                            db_header_columns = db_for_header.get_columns()
444
445                            # Write header file
446                            header_file_tmp = os.path.join(tmpdir, "header")
447                            f = open(header_file_tmp, "w")
448                            vcf.Writer(f, db_header_from_columns)
449                            f.close()
450
451                            # Replace #CHROM line with rel columns
452                            header_list = db_for_header.read_header_file(
453                                header_file=header_file_tmp
454                            )
455                            header_list[-1] = "\t".join(db_header_columns)
456
457                    except:
458
459                        log.warning(
460                            f"No header for file {input_file}. Set as default VCF header"
461                        )
462                        header_list = default_header_list
463
464            else:  # try for unknown format ?
465
466                log.error(f"Input file format '{input_format}' not available")
467                raise ValueError(f"Input file format '{input_format}' not available")
468
469            if not header_list:
470                header_list = default_header_list
471
472            # header as list
473            self.header_list = header_list
474
475            # header as VCF object
476            self.header_vcf = vcf.Reader(io.StringIO("\n".join(header_list)))
477
478        else:
479
480            self.header_list = None
481            self.header_vcf = None

It reads the header of a VCF file and stores it as a list of strings and as a VCF object

def get_query_to_df(self, query: str = '', limit: int = None) -> pandas.core.frame.DataFrame: View Source

483    def get_query_to_df(self, query: str = "", limit: int = None) -> pd.DataFrame:
484        """
485        The `get_query_to_df` function takes a query as a string and returns the result as a pandas
486        DataFrame based on the connection format.
487
488        :param query: The `query` parameter in the `get_query_to_df` function is a string that
489        represents the SQL query you want to execute. This query will be used to fetch data from a
490        database and convert it into a pandas DataFrame
491        :type query: str
492        :param limit: The `limit` parameter in the `get_query_to_df` function is used to specify the
493        maximum number of rows to be returned in the resulting dataframe. If a limit is provided, the
494        function will only fetch up to that number of rows from the database query result. If no limit
495        is specified,
496        :type limit: int
497        :return: A pandas DataFrame is being returned by the `get_query_to_df` function.
498        """
499
500        # Connexion format
501        connexion_format = self.get_connexion_format()
502
503        # Limit in query
504        if limit:
505            pd.set_option("display.max_rows", limit)
506            if connexion_format in ["duckdb"]:
507                df = (
508                    self.conn.execute(query)
509                    .fetch_record_batch(limit)
510                    .read_next_batch()
511                    .to_pandas()
512                )
513            elif connexion_format in ["sqlite"]:
514                df = next(pd.read_sql_query(query, self.conn, chunksize=limit))
515
516        # Full query
517        else:
518            if connexion_format in ["duckdb"]:
519                df = self.conn.execute(query).df()
520            elif connexion_format in ["sqlite"]:
521                df = pd.read_sql_query(query, self.conn)
522
523        return df

The get_query_to_df function takes a query as a string and returns the result as a pandas DataFrame based on the connection format.

Parameters

query: The query parameter in the get_query_to_df function is a string that represents the SQL query you want to execute. This query will be used to fetch data from a database and convert it into a pandas DataFrame
limit: The limit parameter in the get_query_to_df function is used to specify the maximum number of rows to be returned in the resulting dataframe. If a limit is provided, the function will only fetch up to that number of rows from the database query result. If no limit is specified,

Returns

A pandas DataFrame is being returned by the get_query_to_df function.

def get_overview(self) -> None: View Source

525    def get_overview(self) -> None:
526        """
527        The function prints the input, output, config, and dataframe of the current object
528        """
529        table_variants_from = self.get_table_variants(clause="from")
530        sql_columns = self.get_header_columns_as_sql()
531        sql_query_export = f"SELECT {sql_columns} FROM {table_variants_from}"
532        df = self.get_query_to_df(sql_query_export)
533        log.info(
534            "Input:  "
535            + str(self.get_input())
536            + " ["
537            + str(str(self.get_input_format()))
538            + "]"
539        )
540        log.info(
541            "Output: "
542            + str(self.get_output())
543            + " ["
544            + str(str(self.get_output_format()))
545            + "]"
546        )
547        log.info("Config: ")
548        for d in str(json.dumps(self.get_config(), indent=4, sort_keys=True)).split(
549            "\n"
550        ):
551            log.info("\t" + str(d))
552        log.info("Param: ")
553        for d in str(json.dumps(self.get_param(), indent=4, sort_keys=True)).split(
554            "\n"
555        ):
556            log.info("\t" + str(d))
557        log.info("Sample list: " + str(self.get_header_sample_list()))
558        log.info("Dataframe: ")
559        for d in str(df).split("\n"):
560            log.info("\t" + str(d))
561
562        # garbage collector
563        del df
564        gc.collect()
565
566        return None

The function prints the input, output, config, and dataframe of the current object

def get_stats(self) -> dict: View Source

568    def get_stats(self) -> dict:
569        """
570        The `get_stats` function calculates and returns various statistics of the current object,
571        including information about the input file, variants, samples, header fields, quality, and
572        SNVs/InDels.
573        :return: a dictionary containing various statistics of the current object. The dictionary has
574        the following structure:
575        """
576
577        # Log
578        log.info(f"Stats Calculation...")
579
580        # table varaints
581        table_variants_from = self.get_table_variants()
582
583        # stats dict
584        stats = {"Infos": {}}
585
586        ### File
587        input_file = self.get_input()
588        stats["Infos"]["Input file"] = input_file
589
590        # Header
591        header_infos = self.get_header().infos
592        header_formats = self.get_header().formats
593        header_infos_list = list(header_infos)
594        header_formats_list = list(header_formats)
595
596        ### Variants
597
598        stats["Variants"] = {}
599
600        # Variants by chr
601        sql_query_nb_variant_by_chrom = f'SELECT "#CHROM" as CHROM, count(*) as count FROM {table_variants_from} GROUP BY "#CHROM"'
602        df_nb_of_variants_by_chrom = self.get_query_to_df(sql_query_nb_variant_by_chrom)
603        nb_of_variants_by_chrom = df_nb_of_variants_by_chrom.sort_values(
604            by=["CHROM"], kind="quicksort"
605        )
606
607        # Total number of variants
608        nb_of_variants = nb_of_variants_by_chrom["count"].sum()
609
610        # Calculate percentage
611        nb_of_variants_by_chrom["percent"] = nb_of_variants_by_chrom["count"].apply(
612            lambda x: (x / nb_of_variants)
613        )
614
615        stats["Variants"]["Number of variants by chromosome"] = (
616            nb_of_variants_by_chrom.to_dict(orient="index")
617        )
618
619        stats["Infos"]["Number of variants"] = int(nb_of_variants)
620
621        ### Samples
622
623        # Init
624        samples = {}
625        nb_of_samples = 0
626
627        # Check Samples
628        if "GT" in header_formats_list and "FORMAT" in self.get_header_columns():
629            log.debug(f"Check samples...")
630            for sample in self.get_header_sample_list():
631                sql_query_samples = f"""
632                    SELECT  '{sample}' as sample,
633                            REGEXP_EXTRACT("{sample}", '^([0-9/|.]*)[:]*',1) as genotype,
634                            count(REGEXP_EXTRACT("{sample}", '^([0-9/|.]*)[:]*',1)) as count,
635                            concat((count(REGEXP_EXTRACT("{sample}", '^([0-9/|.]*)[:]*',1))/{nb_of_variants})) as percentage
636                    FROM {table_variants_from}
637                    WHERE (
638                        regexp_matches("{sample}", '^[0-9]([/|][0-9])+')
639                        AND
640                        len(string_split(CAST("FORMAT" AS VARCHAR), ':')) = len(string_split(CAST("{sample}" AS VARCHAR), ':'))
641                      )
642                    GROUP BY genotype
643                    """
644                sql_query_genotype_df = self.conn.execute(sql_query_samples).df()
645                sample_genotype_count = sql_query_genotype_df["count"].sum()
646                if len(sql_query_genotype_df):
647                    nb_of_samples += 1
648                    samples[f"{sample} - {sample_genotype_count} variants"] = (
649                        sql_query_genotype_df.to_dict(orient="index")
650                    )
651
652            stats["Samples"] = samples
653            stats["Infos"]["Number of samples"] = nb_of_samples
654
655        # #
656        # if "FORMAT" in self.get_header_columns() and "DP" in header_formats_list:
657        #     stats["Infos"]["Number of samples"] = nb_of_samples
658        # elif nb_of_samples:
659        #     stats["Infos"]["Number of samples"] = "not a VCF format"
660
661        ### INFO and FORMAT fields
662        header_types_df = {}
663        header_types_list = {
664            "List of INFO fields": header_infos,
665            "List of FORMAT fields": header_formats,
666        }
667        i = 0
668        for header_type in header_types_list:
669
670            header_type_infos = header_types_list.get(header_type)
671            header_infos_dict = {}
672
673            for info in header_type_infos:
674
675                i += 1
676                header_infos_dict[i] = {}
677
678                # ID
679                header_infos_dict[i]["id"] = info
680
681                # num
682                genotype_map = {None: ".", -1: "A", -2: "G", -3: "R"}
683                if header_type_infos[info].num in genotype_map.keys():
684                    header_infos_dict[i]["Number"] = genotype_map.get(
685                        header_type_infos[info].num
686                    )
687                else:
688                    header_infos_dict[i]["Number"] = header_type_infos[info].num
689
690                # type
691                if header_type_infos[info].type:
692                    header_infos_dict[i]["Type"] = header_type_infos[info].type
693                else:
694                    header_infos_dict[i]["Type"] = "."
695
696                # desc
697                if header_type_infos[info].desc != None:
698                    header_infos_dict[i]["Description"] = header_type_infos[info].desc
699                else:
700                    header_infos_dict[i]["Description"] = ""
701
702            if len(header_infos_dict):
703                header_types_df[header_type] = pd.DataFrame.from_dict(
704                    header_infos_dict, orient="index"
705                ).to_dict(orient="index")
706
707        # Stats
708        stats["Infos"]["Number of INFO fields"] = len(header_infos_list)
709        stats["Infos"]["Number of FORMAT fields"] = len(header_formats_list)
710        stats["Header"] = header_types_df
711
712        ### QUAL
713        if "QUAL" in self.get_header_columns():
714            sql_query_qual = f"""
715                    SELECT
716                        avg(CAST(QUAL AS INTEGER)) AS Average,
717                        min(CAST(QUAL AS INTEGER)) AS Minimum,
718                        max(CAST(QUAL AS INTEGER)) AS Maximum,
719                        stddev(CAST(QUAL AS INTEGER)) AS StandardDeviation,
720                        median(CAST(QUAL AS INTEGER)) AS Median,
721                        variance(CAST(QUAL AS INTEGER)) AS Variance
722                    FROM {table_variants_from}
723                    WHERE CAST(QUAL AS VARCHAR) NOT IN ('.')
724                    """
725
726            qual = self.conn.execute(sql_query_qual).df().to_dict(orient="index")
727            stats["Quality"] = {"Stats": qual}
728
729        ### SNV and InDel
730
731        sql_query_snv = f"""
732            
733            SELECT Type, count FROM (
734
735                    SELECT
736                        'Total' AS Type,
737                        count(*) AS count
738                    FROM {table_variants_from}
739
740                    UNION
741
742                    SELECT
743                        'MNV' AS Type,
744                        count(*) AS count
745                    FROM {table_variants_from}
746                    WHERE len(REF) > 1 AND len(ALT) > 1
747                    AND len(REF) = len(ALT)
748
749                    UNION
750
751                    SELECT
752                        'InDel' AS Type,
753                        count(*) AS count
754                    FROM {table_variants_from}
755                    WHERE len(REF) > 1 OR len(ALT) > 1
756                    AND len(REF) != len(ALT)
757                    
758                    UNION
759
760                    SELECT
761                        'SNV' AS Type,
762                        count(*) AS count
763                    FROM {table_variants_from}
764                    WHERE len(REF) = 1 AND len(ALT) = 1
765
766                )
767
768            ORDER BY count DESC
769
770                """
771        snv_indel = self.conn.execute(sql_query_snv).df().to_dict(orient="index")
772
773        sql_query_snv_substitution = f"""
774                SELECT
775                    concat(REF, '>', ALT) AS 'Substitution',
776                    count(*) AS count
777                FROM {table_variants_from}
778                WHERE len(REF) = 1 AND len(ALT) = 1
779                GROUP BY REF, ALT
780                ORDER BY count(*) DESC
781                """
782        snv_substitution = (
783            self.conn.execute(sql_query_snv_substitution).df().to_dict(orient="index")
784        )
785        stats["Variants"]["Counts"] = snv_indel
786        stats["Variants"]["Substitutions"] = snv_substitution
787
788        return stats

The get_stats function calculates and returns various statistics of the current object, including information about the input file, variants, samples, header fields, quality, and SNVs/InDels.

Returns

a dictionary containing various statistics of the current object. The dictionary has the following structure:

def stats_to_file(self, file: str = None) -> str: View Source

790    def stats_to_file(self, file: str = None) -> str:
791        """
792        The function `stats_to_file` takes a file name as input, retrieves statistics, serializes them
793        into a JSON object, and writes the JSON object to the specified file.
794
795        :param file: The `file` parameter is a string that represents the file path where the JSON data
796        will be written
797        :type file: str
798        :return: the name of the file that was written to.
799        """
800
801        # Get stats
802        stats = self.get_stats()
803
804        # Serializing json
805        json_object = json.dumps(stats, indent=4)
806
807        # Writing to sample.json
808        with open(file, "w") as outfile:
809            outfile.write(json_object)
810
811        return file

The function stats_to_file takes a file name as input, retrieves statistics, serializes them into a JSON object, and writes the JSON object to the specified file.

Parameters

file: The file parameter is a string that represents the file path where the JSON data will be written

Returns

the name of the file that was written to.

def print_stats(self, output_file: str = None, json_file: str = None) -> None: View Source

813    def print_stats(self, output_file: str = None, json_file: str = None) -> None:
814        """
815        The `print_stats` function generates a markdown file and prints the statistics contained in a
816        JSON file in a formatted manner.
817
818        :param output_file: The `output_file` parameter is a string that specifies the path and filename
819        of the output file where the stats will be printed in Markdown format. If no `output_file` is
820        provided, a temporary directory will be created and the stats will be saved in a file named
821        "stats.md" within that
822        :type output_file: str
823        :param json_file: The `json_file` parameter is a string that represents the path to the JSON
824        file where the statistics will be saved. If no value is provided, a temporary directory will be
825        created and a default file name "stats.json" will be used
826        :type json_file: str
827        :return: The function `print_stats` does not return any value. It has a return type annotation
828        of `None`.
829        """
830
831        # Full path
832        output_file = full_path(output_file)
833        json_file = full_path(json_file)
834
835        with tempfile.TemporaryDirectory() as tmpdir:
836
837            # Files
838            if not output_file:
839                output_file = os.path.join(tmpdir, "stats.md")
840            if not json_file:
841                json_file = os.path.join(tmpdir, "stats.json")
842
843            # Create folders
844            if not os.path.exists(os.path.dirname(output_file)):
845                Path(os.path.dirname(output_file)).mkdir(parents=True, exist_ok=True)
846            if not os.path.exists(os.path.dirname(json_file)):
847                Path(os.path.dirname(json_file)).mkdir(parents=True, exist_ok=True)
848
849            # Create stats JSON file
850            stats_file = self.stats_to_file(file=json_file)
851
852            # Print stats file
853            with open(stats_file) as f:
854                stats = yaml.safe_load(f)
855
856            # Output
857            output_title = []
858            output_index = []
859            output = []
860
861            # Title
862            output_title.append("# HOWARD Stats")
863
864            # Index
865            output_index.append("## Index")
866
867            # Process sections
868            for section in stats:
869                infos = stats.get(section)
870                section_link = "#" + section.lower().replace(" ", "-")
871                output.append(f"## {section}")
872                output_index.append(f"- [{section}]({section_link})")
873
874                if len(infos):
875                    for info in infos:
876                        try:
877                            df = pd.DataFrame.from_dict(infos.get(info), orient="index")
878                            is_df = True
879                        except:
880                            try:
881                                df = pd.DataFrame.from_dict(
882                                    json.loads((infos.get(info))), orient="index"
883                                )
884                                is_df = True
885                            except:
886                                is_df = False
887                        if is_df:
888                            output.append(f"### {info}")
889                            info_link = "#" + info.lower().replace(" ", "-")
890                            output_index.append(f"   - [{info}]({info_link})")
891                            output.append(f"{df.to_markdown(index=False)}")
892                        else:
893                            output.append(f"- {info}: {infos.get(info)}")
894                else:
895                    output.append(f"NA")
896
897            # Write stats in markdown file
898            with open(output_file, "w") as fp:
899                for item in output_title:
900                    fp.write("%s\n" % item)
901                for item in output_index:
902                    fp.write("%s\n" % item)
903                for item in output:
904                    fp.write("%s\n" % item)
905
906            # Output stats in markdown
907            print("")
908            print("\n\n".join(output_title))
909            print("")
910            print("\n\n".join(output))
911            print("")
912
913        return None

The print_stats function generates a markdown file and prints the statistics contained in a JSON file in a formatted manner.

Parameters

output_file: The output_file parameter is a string that specifies the path and filename of the output file where the stats will be printed in Markdown format. If no output_file is provided, a temporary directory will be created and the stats will be saved in a file named "stats.md" within that
json_file: The json_file parameter is a string that represents the path to the JSON file where the statistics will be saved. If no value is provided, a temporary directory will be created and a default file name "stats.json" will be used

Returns

The function print_stats does not return any value. It has a return type annotation of None.

def get_input(self) -> str: View Source

915    def get_input(self) -> str:
916        """
917        It returns the value of the input variable.
918        :return: The input is being returned.
919        """
920        return self.input

It returns the value of the input variable.

Returns

The input is being returned.

def get_input_format(self, input_file: str = None) -> str: View Source

922    def get_input_format(self, input_file: str = None) -> str:
923        """
924        This function returns the format of the input variable, either from the provided input file or
925        by prompting for input.
926
927        :param input_file: The `input_file` parameter in the `get_input_format` method is a string that
928        represents the file path of the input file. If no `input_file` is provided when calling the
929        method, it will default to `None`
930        :type input_file: str
931        :return: The format of the input variable is being returned.
932        """
933
934        if not input_file:
935            input_file = self.get_input()
936        input_format = get_file_format(input_file)
937        return input_format

This function returns the format of the input variable, either from the provided input file or by prompting for input.

Parameters

input_file: The input_file parameter in the get_input_format method is a string that represents the file path of the input file. If no input_file is provided when calling the method, it will default to None

Returns

The format of the input variable is being returned.

def get_input_compressed(self, input_file: str = None) -> str: View Source

939    def get_input_compressed(self, input_file: str = None) -> str:
940        """
941        The function `get_input_compressed` returns the format of the input variable after compressing
942        it.
943
944        :param input_file: The `input_file` parameter in the `get_input_compressed` method is a string
945        that represents the file path of the input file. If no `input_file` is provided when calling the
946        method, it will default to `None` and the method will then call `self.get_input()` to
947        :type input_file: str
948        :return: The function `get_input_compressed` returns the compressed format of the input
949        variable.
950        """
951
952        if not input_file:
953            input_file = self.get_input()
954        input_compressed = get_file_compressed(input_file)
955        return input_compressed

The function get_input_compressed returns the format of the input variable after compressing it.

Parameters

input_file: The input_file parameter in the get_input_compressed method is a string that represents the file path of the input file. If no input_file is provided when calling the method, it will default to None and the method will then call self.get_input() to

Returns

The function get_input_compressed returns the compressed format of the input variable.

def get_output(self) -> str: View Source

957    def get_output(self) -> str:
958        """
959        It returns the output of the neuron.
960        :return: The output of the neural network.
961        """
962
963        return self.output

It returns the output of the neuron.

Returns

The output of the neural network.

def get_output_format(self, output_file: str = None) -> str: View Source

965    def get_output_format(self, output_file: str = None) -> str:
966        """
967        The function `get_output_format` returns the format of the input variable or the output file if
968        provided.
969
970        :param output_file: The `output_file` parameter in the `get_output_format` method is a string
971        that represents the file path of the output file. If no `output_file` is provided when calling
972        the method, it will default to the output obtained from the `get_output` method of the class
973        instance. The
974        :type output_file: str
975        :return: The format of the input variable is being returned.
976        """
977
978        if not output_file:
979            output_file = self.get_output()
980        output_format = get_file_format(output_file)
981
982        return output_format

The function get_output_format returns the format of the input variable or the output file if provided.

Parameters

output_file: The output_file parameter in the get_output_format method is a string that represents the file path of the output file. If no output_file is provided when calling the method, it will default to the output obtained from the get_output method of the class instance. The

Returns

The format of the input variable is being returned.

def get_config(self) -> dict: View Source

984    def get_config(self) -> dict:
985        """
986        It returns the config
987        :return: The config variable is being returned.
988        """
989        return self.config

It returns the config

Returns

The config variable is being returned.

def get_param(self) -> dict: View Source

991    def get_param(self) -> dict:
992        """
993        It returns the param
994        :return: The param variable is being returned.
995        """
996        return self.param

It returns the param

Returns

The param variable is being returned.

def get_connexion_db(self) -> str: View Source

 998    def get_connexion_db(self) -> str:
 999        """
1000        It returns the connexion_db attribute of the object
1001        :return: The connexion_db is being returned.
1002        """
1003        return self.connexion_db

It returns the connexion_db attribute of the object

Returns

The connexion_db is being returned.

def get_prefix(self) -> str: View Source

1005    def get_prefix(self) -> str:
1006        """
1007        It returns the prefix of the object.
1008        :return: The prefix is being returned.
1009        """
1010        return self.prefix

It returns the prefix of the object.

Returns

The prefix is being returned.

def get_table_variants(self, clause: str = 'select') -> str: View Source

1012    def get_table_variants(self, clause: str = "select") -> str:
1013        """
1014        This function returns the table_variants attribute of the object
1015
1016        :param clause: the type of clause the table will be used. Either "select" or "from" (optional),
1017        defaults to select (optional)
1018        :return: The table_variants attribute of the object.
1019        """
1020
1021        # Access
1022        access = self.get_config().get("access", None)
1023
1024        # Clauses "select", "where", "update"
1025        if clause in ["select", "where", "update"]:
1026            table_variants = self.table_variants
1027        # Clause "from"
1028        elif clause in ["from"]:
1029            # For Read Only
1030            if self.get_input_format() in ["parquet"] and access in ["RO"]:
1031                input_file = self.get_input()
1032                table_variants = f"'{input_file}' as variants"
1033            # For Read Write
1034            else:
1035                table_variants = f"{self.table_variants} as variants"
1036        else:
1037            table_variants = self.table_variants
1038        return table_variants

This function returns the table_variants attribute of the object

Parameters

clause: the type of clause the table will be used. Either "select" or "from" (optional), defaults to select (optional)

Returns

The table_variants attribute of the object.

def get_tmp_dir(self) -> str: View Source

1040    def get_tmp_dir(self) -> str:
1041        """
1042        The function `get_tmp_dir` returns the temporary directory path based on configuration
1043        parameters or a default path.
1044        :return: The `get_tmp_dir` method is returning the temporary directory path based on the
1045        configuration, parameters, and a default value of "/tmp".
1046        """
1047
1048        return get_tmp(
1049            config=self.get_config(), param=self.get_param(), default_tmp="/tmp"
1050        )

The function get_tmp_dir returns the temporary directory path based on configuration parameters or a default path.

Returns

The get_tmp_dir method is returning the temporary directory path based on the configuration, parameters, and a default value of "/tmp".

def get_connexion_type(self) -> str: View Source

1052    def get_connexion_type(self) -> str:
1053        """
1054        If the connexion type is not in the list of allowed connexion types, raise a ValueError
1055
1056        :return: The connexion type is being returned.
1057        """
1058        return self.get_config().get("connexion_type", "memory")

If the connexion type is not in the list of allowed connexion types, raise a ValueError

Returns

The connexion type is being returned.

def get_connexion(self): View Source

1060    def get_connexion(self):
1061        """
1062        It returns the connection object
1063
1064        :return: The connection object.
1065        """
1066        return self.conn

It returns the connection object

Returns

The connection object.

def close_connexion(self) -> None: View Source

1068    def close_connexion(self) -> None:
1069        """
1070        This function closes the connection to the database.
1071        :return: The connection is being closed.
1072        """
1073        return self.conn.close()

This function closes the connection to the database.

Returns

The connection is being closed.

def get_header(self, type: str = 'vcf'): View Source

1075    def get_header(self, type: str = "vcf"):
1076        """
1077        This function returns the header of the VCF file as a list of strings
1078
1079        :param type: the type of header you want to get, defaults to vcf (optional)
1080        :return: The header of the vcf file.
1081        """
1082
1083        if self.header_vcf:
1084            if type == "vcf":
1085                return self.header_vcf
1086            elif type == "list":
1087                return self.header_list
1088        else:
1089            if type == "vcf":
1090                header = vcf.Reader(io.StringIO("\n".join(vcf_required)))
1091                return header
1092            elif type == "list":
1093                return vcf_required

This function returns the header of the VCF file as a list of strings

Parameters

type: the type of header you want to get, defaults to vcf (optional)

Returns

The header of the vcf file.

def get_header_length(self, file: str = None) -> int: View Source

1095    def get_header_length(self, file: str = None) -> int:
1096        """
1097        The function `get_header_length` returns the length of the header list, excluding the #CHROM
1098        line.
1099
1100        :param file: The `file` parameter is an optional argument that specifies the path to a VCF
1101        header file. If this argument is provided, the function will read the header from the specified
1102        file and return the length of the header list minus 1 (to exclude the #CHROM line)
1103        :type file: str
1104        :return: the length of the header list, excluding the #CHROM line.
1105        """
1106
1107        if file:
1108            return len(self.read_vcf_header_file(file=file)) - 1
1109        elif self.get_header(type="list"):
1110            return len(self.get_header(type="list")) - 1
1111        else:
1112            return 0

The function get_header_length returns the length of the header list, excluding the #CHROM line.

Parameters

file: The file parameter is an optional argument that specifies the path to a VCF header file. If this argument is provided, the function will read the header from the specified file and return the length of the header list minus 1 (to exclude the #CHROM line)

Returns

the length of the header list, excluding the #CHROM line.

def get_header_columns(self) -> str: View Source

1114    def get_header_columns(self) -> str:
1115        """
1116        This function returns the header list of a VCF
1117
1118        :return: The length of the header list.
1119        """
1120        if self.get_header():
1121            return self.get_header(type="list")[-1]
1122        else:
1123            return ""

This function returns the header list of a VCF

Returns

The length of the header list.

def get_header_columns_as_list(self) -> list: View Source

1125    def get_header_columns_as_list(self) -> list:
1126        """
1127        This function returns the header list of a VCF
1128
1129        :return: The length of the header list.
1130        """
1131        if self.get_header():
1132            return self.get_header_columns().strip().split("\t")
1133        else:
1134            return []

This function returns the header list of a VCF

Returns

The length of the header list.

def get_header_columns_as_sql(self) -> str: View Source

1136    def get_header_columns_as_sql(self) -> str:
1137        """
1138        This function retruns header length (without #CHROM line)
1139
1140        :return: The length of the header list.
1141        """
1142        sql_column_list = []
1143        for col in self.get_header_columns_as_list():
1144            sql_column_list.append(f'"{col}"')
1145        return ",".join(sql_column_list)

This function retruns header length (without #CHROM line)

Returns

The length of the header list.

def get_header_sample_list( self, check: bool = False, samples: list = None, samples_force: bool = False) -> list: View Source

1147    def get_header_sample_list(
1148        self, check: bool = False, samples: list = None, samples_force: bool = False
1149    ) -> list:
1150        """
1151        The function `get_header_sample_list` returns a list of samples from a VCF header, with optional
1152        checking and filtering based on input parameters.
1153
1154        :param check: The `check` parameter in the `get_header_sample_list` function is a boolean
1155        parameter that determines whether to check if the samples in the list are properly defined as
1156        genotype columns. If `check` is set to `True`, the function will verify if each sample in the
1157        list is defined as a, defaults to False
1158        :type check: bool (optional)
1159        :param samples: The `samples` parameter in the `get_header_sample_list` function is a list that
1160        allows you to specify a subset of samples from the header. If you provide a list of sample
1161        names, the function will check if each sample is defined in the header. If a sample is not found
1162        in the
1163        :type samples: list
1164        :param samples_force: The `samples_force` parameter in the `get_header_sample_list` function is
1165        a boolean parameter that determines whether to force the function to return the sample list
1166        without checking if the samples are genotype columns. If `samples_force` is set to `True`, the
1167        function will return the sample list without performing, defaults to False
1168        :type samples_force: bool (optional)
1169        :return: The function `get_header_sample_list` returns a list of samples based on the input
1170        parameters and conditions specified in the function.
1171        """
1172
1173        # Init
1174        samples_list = []
1175
1176        if samples is None:
1177            samples_list = self.header_vcf.samples
1178        else:
1179            samples_checked = []
1180            for sample in samples:
1181                if sample in self.header_vcf.samples:
1182                    samples_checked.append(sample)
1183                else:
1184                    log.warning(f"Sample '{sample}' not defined in header")
1185            samples_list = samples_checked
1186
1187            # Force sample list without checking if is_genotype_column
1188            if samples_force:
1189                log.warning(f"Samples {samples_list} not checked if genotypes")
1190                return samples_list
1191
1192        if check:
1193            samples_checked = []
1194            for sample in samples_list:
1195                if self.is_genotype_column(column=sample):
1196                    samples_checked.append(sample)
1197                else:
1198                    log.warning(
1199                        f"Sample '{sample}' not defined as a sample (genotype not well defined)"
1200                    )
1201            samples_list = samples_checked
1202
1203        # Return samples list
1204        return samples_list

The function get_header_sample_list returns a list of samples from a VCF header, with optional checking and filtering based on input parameters.

Parameters

check: The check parameter in the get_header_sample_list function is a boolean parameter that determines whether to check if the samples in the list are properly defined as genotype columns. If check is set to True, the function will verify if each sample in the list is defined as a, defaults to False
samples: The samples parameter in the get_header_sample_list function is a list that allows you to specify a subset of samples from the header. If you provide a list of sample names, the function will check if each sample is defined in the header. If a sample is not found in the
samples_force: The samples_force parameter in the get_header_sample_list function is a boolean parameter that determines whether to force the function to return the sample list without checking if the samples are genotype columns. If samples_force is set to True, the function will return the sample list without performing, defaults to False

Returns

The function get_header_sample_list returns a list of samples based on the input parameters and conditions specified in the function.

def is_genotype_column(self, column: str = None) -> bool: View Source

1206    def is_genotype_column(self, column: str = None) -> bool:
1207        """
1208        This function checks if a given column is a genotype column in a database.
1209
1210        :param column: The `column` parameter in the `is_genotype_column` method is a string that
1211        represents the column name in a database table. This method checks if the specified column is a
1212        genotype column in the database. If a column name is provided, it calls the `is_genotype_column`
1213        method of
1214        :type column: str
1215        :return: The `is_genotype_column` method is returning a boolean value. If the `column` parameter
1216        is not None, it calls the `is_genotype_column` method of the `Database` class with the specified
1217        column name and returns the result. If the `column` parameter is None, it returns False.
1218        """
1219
1220        if column is not None:
1221            return Database(database=self.get_input()).is_genotype_column(column=column)
1222        else:
1223            return False

This function checks if a given column is a genotype column in a database.

Parameters

column: The column parameter in the is_genotype_column method is a string that represents the column name in a database table. This method checks if the specified column is a genotype column in the database. If a column name is provided, it calls the is_genotype_column method of

Returns

The is_genotype_column method is returning a boolean value. If the column parameter is not None, it calls the is_genotype_column method of the Database class with the specified column name and returns the result. If the column parameter is None, it returns False.

def get_verbose(self) -> bool: View Source

1225    def get_verbose(self) -> bool:
1226        """
1227        It returns the value of the "verbose" key in the config dictionary, or False if the key doesn't
1228        exist
1229
1230        :return: The value of the key "verbose" in the config dictionary.
1231        """
1232        return self.get_config().get("verbose", False)

It returns the value of the "verbose" key in the config dictionary, or False if the key doesn't exist

Returns

The value of the key "verbose" in the config dictionary.

def get_connexion_format(self) -> str: View Source

1234    def get_connexion_format(self) -> str:
1235        """
1236        It returns the connexion format of the object.
1237        :return: The connexion_format is being returned.
1238        """
1239        connexion_format = self.connexion_format
1240        if connexion_format not in ["duckdb", "sqlite"]:
1241            log.error(f"Unknown connexion format {connexion_format}")
1242            raise ValueError(f"Unknown connexion format {connexion_format}")
1243        else:
1244            return connexion_format

It returns the connexion format of the object.

Returns

The connexion_format is being returned.

def insert_file_to_table( self, file, columns: str, header_len: int = 0, sep: str = '\t', chunksize: int = 1000000) -> None: View Source

1246    def insert_file_to_table(
1247        self,
1248        file,
1249        columns: str,
1250        header_len: int = 0,
1251        sep: str = "\t",
1252        chunksize: int = 1000000,
1253    ) -> None:
1254        """
1255        The function reads a file in chunks and inserts each chunk into a table based on the specified
1256        database format.
1257
1258        :param file: The `file` parameter is the file that you want to load into a table. It should be
1259        the path to the file on your system
1260        :param columns: The `columns` parameter in the `insert_file_to_table` function is a string that
1261        should contain the names of the columns in the table where the data will be inserted. The column
1262        names should be separated by commas within the string. For example, if you have columns named
1263        "id", "name
1264        :type columns: str
1265        :param header_len: The `header_len` parameter in the `insert_file_to_table` function specifies
1266        the number of lines to skip at the beginning of the file before reading the actual data. This
1267        parameter allows you to skip any header information present in the file before processing the
1268        data, defaults to 0
1269        :type header_len: int (optional)
1270        :param sep: The `sep` parameter in the `insert_file_to_table` function is used to specify the
1271        separator character that is used in the file being read. In this case, the default separator is
1272        set to `\t`, which represents a tab character. You can change this parameter to a different
1273        separator character if, defaults to \t
1274        :type sep: str (optional)
1275        :param chunksize: The `chunksize` parameter specifies the number of rows to read in at a time
1276        when processing the file in chunks. In the provided code snippet, the default value for
1277        `chunksize` is set to 1000000. This means that the file will be read in chunks of 1,, defaults
1278        to 1000000
1279        :type chunksize: int (optional)
1280        """
1281
1282        # Config
1283        chunksize = self.get_config().get("load", {}).get("chunk", chunksize)
1284        connexion_format = self.get_connexion_format()
1285
1286        log.debug("chunksize: " + str(chunksize))
1287
1288        if chunksize:
1289            for chunk in pd.read_csv(
1290                file, skiprows=header_len, sep=sep, chunksize=chunksize, engine="c"
1291            ):
1292                if connexion_format in ["duckdb"]:
1293                    sql_insert_into = (
1294                        f"INSERT INTO variants ({columns}) SELECT {columns} FROM chunk"
1295                    )
1296                    self.conn.execute(sql_insert_into)
1297                elif connexion_format in ["sqlite"]:
1298                    chunk.to_sql("variants", self.conn, if_exists="append", index=False)

The function reads a file in chunks and inserts each chunk into a table based on the specified database format.

Parameters

file: The file parameter is the file that you want to load into a table. It should be the path to the file on your system
columns: The columns parameter in the insert_file_to_table function is a string that should contain the names of the columns in the table where the data will be inserted. The column names should be separated by commas within the string. For example, if you have columns named "id", "name
header_len: The header_len parameter in the insert_file_to_table function specifies the number of lines to skip at the beginning of the file before reading the actual data. This parameter allows you to skip any header information present in the file before processing the data, defaults to 0
sep: The sep parameter in the insert_file_to_table function is used to specify the separator character that is used in the file being read. In this case, the default separator is set to , which represents a tab character. You can change this parameter to a different separator character if, defaults to
chunksize: The chunksize parameter specifies the number of rows to read in at a time when processing the file in chunks. In the provided code snippet, the default value for chunksize is set to 1000000. This means that the file will be read in chunks of 1,, defaults to 1000000

def load_data( self, input_file: str = None, drop_variants_table: bool = False, sample_size: int = 20480) -> None: View Source

1300    def load_data(
1301        self,
1302        input_file: str = None,
1303        drop_variants_table: bool = False,
1304        sample_size: int = 20480,
1305    ) -> None:
1306        """
1307        The `load_data` function reads a VCF file and inserts it into a table, with options to drop the
1308        table before loading the data and specify a sample size.
1309
1310        :param input_file: The path to the input file. This is the VCF file that will be loaded into the
1311        table
1312        :type input_file: str
1313        :param drop_variants_table: The `drop_variants_table` parameter is a boolean flag that
1314        determines whether the variants table should be dropped before loading the data. If set to
1315        `True`, the variants table will be dropped. If set to `False` (default), the variants table will
1316        not be dropped, defaults to False
1317        :type drop_variants_table: bool (optional)
1318        :param sample_size: The `sample_size` parameter determines the number of rows to be sampled from
1319        the input file. If it is set to `None`, the default value of 20480 will be used, defaults to
1320        20480
1321        :type sample_size: int (optional)
1322        """
1323
1324        log.info("Loading...")
1325
1326        # change input file
1327        if input_file:
1328            self.set_input(input_file)
1329            self.set_header()
1330
1331        # drop variants table
1332        if drop_variants_table:
1333            self.drop_variants_table()
1334
1335        # get table variants
1336        table_variants = self.get_table_variants()
1337
1338        # Access
1339        access = self.get_config().get("access", None)
1340        log.debug(f"access: {access}")
1341
1342        # Input format and compress
1343        input_format = self.get_input_format()
1344        input_compressed = self.get_input_compressed()
1345        log.debug(f"input_format: {input_format}")
1346        log.debug(f"input_compressed: {input_compressed}")
1347
1348        # input_compressed_format
1349        if input_compressed:
1350            input_compressed_format = "gzip"
1351        else:
1352            input_compressed_format = "none"
1353        log.debug(f"input_compressed_format: {input_compressed_format}")
1354
1355        # Connexion format
1356        connexion_format = self.get_connexion_format()
1357
1358        # Sample size
1359        if not sample_size:
1360            sample_size = -1
1361        log.debug(f"sample_size: {sample_size}")
1362
1363        # Load data
1364        log.debug(f"Load Data from {input_format}")
1365
1366        # DuckDB connexion
1367        if connexion_format in ["duckdb"]:
1368
1369            # Database already exists
1370            if self.input_format in ["db", "duckdb"]:
1371
1372                if connexion_format in ["duckdb"]:
1373                    log.debug(f"Input file format '{self.input_format}' duckDB")
1374                else:
1375                    log.error(
1376                        f"Input file format '{self.input_format}' not compatilbe with database format '{connexion_format}'"
1377                    )
1378                    raise ValueError(
1379                        f"Input file format '{self.input_format}' not compatilbe with database format '{connexion_format}'"
1380                    )
1381
1382            # Load from existing database format
1383            else:
1384
1385                try:
1386                    # Create Table or View
1387                    database = Database(database=self.input)
1388                    sql_from = database.get_sql_from(sample_size=sample_size)
1389
1390                    if access in ["RO"]:
1391                        sql_load = (
1392                            f"CREATE VIEW {table_variants} AS SELECT * FROM {sql_from}"
1393                        )
1394                    else:
1395                        sql_load = (
1396                            f"CREATE TABLE {table_variants} AS SELECT * FROM {sql_from}"
1397                        )
1398                    self.conn.execute(sql_load)
1399
1400                except:
1401                    # Format not available
1402                    log.error(f"Input file format '{self.input_format}' not available")
1403                    raise ValueError(
1404                        f"Input file format '{self.input_format}' not available"
1405                    )
1406
1407        # SQLite connexion
1408        elif connexion_format in ["sqlite"] and input_format in [
1409            "vcf",
1410            "tsv",
1411            "csv",
1412            "psv",
1413        ]:
1414
1415            # Main structure
1416            structure = {
1417                "#CHROM": "VARCHAR",
1418                "POS": "INTEGER",
1419                "ID": "VARCHAR",
1420                "REF": "VARCHAR",
1421                "ALT": "VARCHAR",
1422                "QUAL": "VARCHAR",
1423                "FILTER": "VARCHAR",
1424                "INFO": "VARCHAR",
1425            }
1426
1427            # Strcuture with samples
1428            structure_complete = structure
1429            if self.get_header_sample_list():
1430                structure["FORMAT"] = "VARCHAR"
1431                for sample in self.get_header_sample_list():
1432                    structure_complete[sample] = "VARCHAR"
1433
1434            # Columns list for create and insert
1435            sql_create_table_columns = []
1436            sql_create_table_columns_list = []
1437            for column in structure_complete:
1438                column_type = structure_complete[column]
1439                sql_create_table_columns.append(
1440                    f'"{column}" {column_type} default NULL'
1441                )
1442                sql_create_table_columns_list.append(f'"{column}"')
1443
1444            # Create database
1445            log.debug(f"Create Table {table_variants}")
1446            sql_create_table_columns_sql = ", ".join(sql_create_table_columns)
1447            sql_create_table_columns_list_sql = ", ".join(sql_create_table_columns_list)
1448            sql_create_table = f"CREATE TABLE IF NOT EXISTS {table_variants} ({sql_create_table_columns_sql})"
1449            self.conn.execute(sql_create_table)
1450
1451            # chunksize define length of file chunk load file
1452            chunksize = 100000
1453
1454            # delimiter
1455            delimiter = file_format_delimiters.get(input_format, "\t")
1456
1457            # Load the input file
1458            with open(self.input, "rt") as input_file:
1459
1460                # Use the appropriate file handler based on the input format
1461                if input_compressed:
1462                    input_file = bgzf.open(self.input, "rt")
1463                if input_format in ["vcf"]:
1464                    header_len = self.get_header_length()
1465                else:
1466                    header_len = 0
1467
1468                # Insert the file contents into a table
1469                self.insert_file_to_table(
1470                    input_file,
1471                    columns=sql_create_table_columns_list_sql,
1472                    header_len=header_len,
1473                    sep=delimiter,
1474                    chunksize=chunksize,
1475                )
1476
1477        else:
1478            log.error(
1479                f"Connexion format '{connexion_format}' not available with format '{input_format}'"
1480            )
1481            raise ValueError(
1482                f"Connexion format '{connexion_format}' not available with format '{input_format}'"
1483            )
1484
1485        # Explode INFOS fields into table fields
1486        if self.get_explode_infos():
1487            self.explode_infos(
1488                prefix=self.get_explode_infos_prefix(),
1489                fields=self.get_explode_infos_fields(),
1490                force=True,
1491            )
1492
1493        # Create index after insertion
1494        self.create_indexes()

The load_data function reads a VCF file and inserts it into a table, with options to drop the table before loading the data and specify a sample size.

Parameters

input_file: The path to the input file. This is the VCF file that will be loaded into the table
drop_variants_table: The drop_variants_table parameter is a boolean flag that determines whether the variants table should be dropped before loading the data. If set to True, the variants table will be dropped. If set to False (default), the variants table will not be dropped, defaults to False
sample_size: The sample_size parameter determines the number of rows to be sampled from the input file. If it is set to None, the default value of 20480 will be used, defaults to 20480

def get_explode_infos(self) -> bool: View Source

1496    def get_explode_infos(self) -> bool:
1497        """
1498        The function `get_explode_infos` returns the value of the "explode_infos" parameter, defaulting
1499        to False if it is not set.
1500        :return: The method is returning the value of the "explode_infos" parameter, which is a boolean
1501        value. If the parameter is not present, it will return False.
1502        """
1503
1504        return self.get_param().get("explode", {}).get("explode_infos", False)

The function get_explode_infos returns the value of the "explode_infos" parameter, defaulting to False if it is not set.

Returns

The method is returning the value of the "explode_infos" parameter, which is a boolean value. If the parameter is not present, it will return False.

def get_explode_infos_fields( self, explode_infos_fields: str = None, remove_fields_not_in_header: bool = False) -> list: View Source

1506    def get_explode_infos_fields(
1507        self,
1508        explode_infos_fields: str = None,
1509        remove_fields_not_in_header: bool = False,
1510    ) -> list:
1511        """
1512        The `get_explode_infos_fields` function returns a list of exploded information fields based on
1513        the input parameter `explode_infos_fields`.
1514
1515        :param explode_infos_fields: The `explode_infos_fields` parameter is a string that specifies the
1516        fields to be exploded. It can be set to "ALL" to explode all fields, or it can be a
1517        comma-separated list of field names to explode
1518        :type explode_infos_fields: str
1519        :param remove_fields_not_in_header: The parameter `remove_fields_not_in_header` is a boolean
1520        flag that determines whether to remove fields that are not present in the header. If it is set
1521        to `True`, any field that is not in the header will be excluded from the list of exploded
1522        information fields. If it is set to `, defaults to False
1523        :type remove_fields_not_in_header: bool (optional)
1524        :return: The function `get_explode_infos_fields` returns a list of exploded information fields.
1525        If the `explode_infos_fields` parameter is not provided or is set to None, it returns an empty
1526        list. If the parameter is provided and its value is "ALL", it also returns an empty list.
1527        Otherwise, it returns a list of exploded information fields after removing any spaces and
1528        splitting the string by commas.
1529        """
1530
1531        # If no fields, get it in param
1532        if not explode_infos_fields:
1533            explode_infos_fields = (
1534                self.get_param().get("explode", {}).get("explode_infos_fields", None)
1535            )
1536
1537        # If no fields, defined as all fields in header using keyword
1538        if not explode_infos_fields:
1539            explode_infos_fields = "*"
1540
1541        # If fields list not empty
1542        if explode_infos_fields:
1543
1544            # Input fields list
1545            if isinstance(explode_infos_fields, str):
1546                fields_input = explode_infos_fields.split(",")
1547            elif isinstance(explode_infos_fields, list):
1548                fields_input = explode_infos_fields
1549            else:
1550                fields_input = []
1551
1552            # Fields list without * keyword
1553            fields_without_all = fields_input.copy()
1554            if "*".casefold() in (item.casefold() for item in fields_without_all):
1555                fields_without_all.remove("*")
1556
1557            # Fields in header
1558            fields_in_header = sorted(list(set(self.get_header().infos)))
1559
1560            # Construct list of fields
1561            fields_output = []
1562            for field in fields_input:
1563
1564                # Strip field
1565                field = field.strip()
1566
1567                # format keyword * in regex
1568                if field.upper() in ["*"]:
1569                    field = ".*"
1570
1571                # Find all fields with pattern
1572                r = re.compile(field)
1573                fields_search = sorted(list(filter(r.match, fields_in_header)))
1574
1575                # Remove fields input from search
1576                if field in fields_search:
1577                    fields_search = [field]
1578                elif fields_search != [field]:
1579                    fields_search = sorted(
1580                        list(set(fields_search).difference(fields_input))
1581                    )
1582
1583                # If field is not in header (avoid not well formatted header)
1584                if not fields_search and not remove_fields_not_in_header:
1585                    fields_search = [field]
1586
1587                # Add found fields
1588                for new_field in fields_search:
1589                    # Add field, if not already exists, and if it is in header (if asked)
1590                    if (
1591                        new_field not in fields_output
1592                        and (
1593                            not remove_fields_not_in_header
1594                            or new_field in fields_in_header
1595                        )
1596                        and new_field not in [".*"]
1597                    ):
1598                        fields_output.append(new_field)
1599
1600            return fields_output
1601
1602        else:
1603
1604            return []

The get_explode_infos_fields function returns a list of exploded information fields based on the input parameter explode_infos_fields.

Parameters

explode_infos_fields: The explode_infos_fields parameter is a string that specifies the fields to be exploded. It can be set to "ALL" to explode all fields, or it can be a comma-separated list of field names to explode
remove_fields_not_in_header: The parameter remove_fields_not_in_header is a boolean flag that determines whether to remove fields that are not present in the header. If it is set to True, any field that is not in the header will be excluded from the list of exploded information fields. If it is set to `, defaults to False

Returns

The function get_explode_infos_fields returns a list of exploded information fields. If the explode_infos_fields parameter is not provided or is set to None, it returns an empty list. If the parameter is provided and its value is "ALL", it also returns an empty list. Otherwise, it returns a list of exploded information fields after removing any spaces and splitting the string by commas.

def get_explode_infos_prefix(self, explode_infos_prefix: str = None) -> str: View Source

1606    def get_explode_infos_prefix(self, explode_infos_prefix: str = None) -> str:
1607        """
1608        The function `get_explode_infos_prefix` returns the value of the `explode_infos_prefix` parameter, or
1609        the value of `self.get_param().get("explode_infos_prefix", None)` if `explode_infos_prefix` is
1610        not provided.
1611
1612        :param explode_infos_prefix: The parameter `explode_infos_prefix` is a string that specifies a
1613        prefix to be used for exploding or expanding information
1614        :type explode_infos_prefix: str
1615        :return: the value of the variable `explode_infos_prefix`.
1616        """
1617
1618        if not explode_infos_prefix:
1619            explode_infos_prefix = (
1620                self.get_param().get("explode", {}).get("explode_infos_prefix", "")
1621            )
1622
1623        return explode_infos_prefix

The function get_explode_infos_prefix returns the value of the explode_infos_prefix parameter, or the value of self.get_param().get("explode_infos_prefix", None) if explode_infos_prefix is not provided.

Parameters

explode_infos_prefix: The parameter explode_infos_prefix is a string that specifies a prefix to be used for exploding or expanding information

Returns

the value of the variable explode_infos_prefix.

def add_column( self, table_name, column_name, column_type, default_value=None, drop: bool = False) -> dict: View Source

1625    def add_column(
1626        self,
1627        table_name,
1628        column_name,
1629        column_type,
1630        default_value=None,
1631        drop: bool = False,
1632    ) -> dict:
1633        """
1634        The `add_column` function adds a column to a SQLite or DuckDB table with a default value if it
1635        doesn't already exist.
1636
1637        :param table_name: The name of the table to which you want to add a column
1638        :param column_name: The parameter "column_name" is the name of the column that you want to add
1639        to the table
1640        :param column_type: The `column_type` parameter specifies the data type of the column that you
1641        want to add to the table. It should be a string that represents the desired data type, such as
1642        "INTEGER", "TEXT", "REAL", etc
1643        :param default_value: The `default_value` parameter is an optional parameter that specifies the
1644        default value for the newly added column. If a default value is provided, it will be assigned to
1645        the column for any existing rows that do not have a value for that column
1646        :param drop: The `drop` parameter is a boolean flag that determines whether to drop the column
1647        if it already exists in the table. If `drop` is set to `True`, the function will drop the
1648        existing column before adding the new column. If `drop` is set to `False` (default),, defaults
1649        to False
1650        :type drop: bool (optional)
1651        :return: a boolean value indicating whether the column was successfully added to the table.
1652        """
1653
1654        # added
1655        added = False
1656        dropped = False
1657
1658        # Check if the column already exists in the table
1659        query = f""" SELECT * FROM {table_name} LIMIT 0 """
1660        columns = self.get_query_to_df(query).columns.tolist()
1661        if column_name.upper() in [c.upper() for c in columns]:
1662            log.debug(
1663                f"The {column_name} column already exists in the {table_name} table"
1664            )
1665            if drop:
1666                self.drop_column(table_name=table_name, column_name=column_name)
1667                dropped = True
1668            else:
1669                return None
1670        else:
1671            log.debug(f"The {column_name} column NOT exists in the {table_name} table")
1672
1673        # Add column in table
1674        add_column_query = (
1675            f""" ALTER TABLE {table_name} ADD COLUMN "{column_name}" {column_type} """
1676        )
1677        if default_value is not None:
1678            add_column_query += f" DEFAULT {default_value}"
1679        self.execute_query(add_column_query)
1680        added = not dropped
1681        log.debug(
1682            f"The {column_name} column was successfully added to the {table_name} table"
1683        )
1684
1685        if added:
1686            added_column = {
1687                "table_name": table_name,
1688                "column_name": column_name,
1689                "column_type": column_type,
1690                "default_value": default_value,
1691            }
1692        else:
1693            added_column = None
1694
1695        return added_column

The add_column function adds a column to a SQLite or DuckDB table with a default value if it doesn't already exist.

Parameters

table_name: The name of the table to which you want to add a column
column_name: The parameter "column_name" is the name of the column that you want to add to the table
column_type: The column_type parameter specifies the data type of the column that you want to add to the table. It should be a string that represents the desired data type, such as "INTEGER", "TEXT", "REAL", etc
default_value: The default_value parameter is an optional parameter that specifies the default value for the newly added column. If a default value is provided, it will be assigned to the column for any existing rows that do not have a value for that column
drop: The drop parameter is a boolean flag that determines whether to drop the column if it already exists in the table. If drop is set to True, the function will drop the existing column before adding the new column. If drop is set to False (default),, defaults to False

Returns

a boolean value indicating whether the column was successfully added to the table.

def drop_column( self, column: dict = None, table_name: str = None, column_name: str = None) -> bool: View Source

1697    def drop_column(
1698        self, column: dict = None, table_name: str = None, column_name: str = None
1699    ) -> bool:
1700        """
1701        The `drop_column` function drops a specified column from a given table in a database and returns
1702        True if the column was successfully dropped, and False if the column does not exist in the
1703        table.
1704
1705        :param column: The `column` parameter is a dictionary that contains information about the column
1706        you want to drop. It has two keys:
1707        :type column: dict
1708        :param table_name: The `table_name` parameter is the name of the table from which you want to
1709        drop a column
1710        :type table_name: str
1711        :param column_name: The `column_name` parameter is the name of the column that you want to drop
1712        from the table
1713        :type column_name: str
1714        :return: a boolean value. It returns True if the column was successfully dropped from the table,
1715        and False if the column does not exist in the table.
1716        """
1717
1718        # Find column infos
1719        if column:
1720            if isinstance(column, dict):
1721                table_name = column.get("table_name", None)
1722                column_name = column.get("column_name", None)
1723            elif isinstance(column, str):
1724                table_name = self.get_table_variants()
1725                column_name = column
1726            else:
1727                table_name = None
1728                column_name = None
1729
1730        if not table_name and not column_name:
1731            return False
1732
1733        # Removed
1734        removed = False
1735
1736        # Check if the column already exists in the table
1737        query = f""" SELECT * FROM {table_name} LIMIT 0 """
1738        columns = self.get_query_to_df(query).columns.tolist()
1739        if column_name in columns:
1740            log.debug(f"The {column_name} column exists in the {table_name} table")
1741        else:
1742            log.debug(f"The {column_name} column NOT exists in the {table_name} table")
1743            return False
1744
1745        # Add column in table # ALTER TABLE integers DROP k
1746        add_column_query = f""" ALTER TABLE {table_name} DROP "{column_name}" """
1747        self.execute_query(add_column_query)
1748        removed = True
1749        log.debug(
1750            f"The {column_name} column was successfully dropped to the {table_name} table"
1751        )
1752
1753        return removed

The drop_column function drops a specified column from a given table in a database and returns True if the column was successfully dropped, and False if the column does not exist in the table.

Parameters

column: The column parameter is a dictionary that contains information about the column you want to drop. It has two keys:
table_name: The table_name parameter is the name of the table from which you want to drop a column
column_name: The column_name parameter is the name of the column that you want to drop from the table

Returns

a boolean value. It returns True if the column was successfully dropped from the table, and False if the column does not exist in the table.

def explode_infos( self, prefix: str = None, create_index: bool = False, fields: list = None, force: bool = False, proccess_all_fields_together: bool = False, table: str = None) -> list: View Source

1755    def explode_infos(
1756        self,
1757        prefix: str = None,
1758        create_index: bool = False,
1759        fields: list = None,
1760        force: bool = False,
1761        proccess_all_fields_together: bool = False,
1762        table: str = None,
1763    ) -> list:
1764        """
1765        The `explode_infos` function in Python takes a VCF file and explodes the INFO fields into
1766        individual columns, returning a list of added columns.
1767
1768        :param prefix: The `prefix` parameter is a string that is used as a prefix for the exploded INFO
1769        fields. If the `prefix` is not provided or is set to `None`, the function will use the value of
1770        `self.get_explode_infos_prefix()` as the prefix
1771        :type prefix: str
1772        :param create_index: The `create_index` parameter is a boolean flag that specifies whether to
1773        create indexes on the exploded INFO fields. If set to `True`, indexes will be created; if set to
1774        `False`, indexes will not be created. The default value is `False`, defaults to False
1775        :type create_index: bool (optional)
1776        :param fields: The `fields` parameter in the `explode_infos` function is a list of INFO fields
1777        that you want to explode into individual columns. If this parameter is not provided, all INFO
1778        fields will be exploded. You can specify the INFO fields you want to explode by passing them as
1779        a list to the `
1780        :type fields: list
1781        :param force: The `force` parameter in the `explode_infos` function is a boolean flag that
1782        determines whether to drop and recreate a column if it already exists in the table. If `force`
1783        is set to `True`, the column will be dropped and recreated. If `force` is set to `False,
1784        defaults to False
1785        :type force: bool (optional)
1786        :param proccess_all_fields_together: The `proccess_all_fields_together` parameter is a boolean
1787        flag that determines whether to process all the INFO fields together or individually. If set to
1788        `True`, all the INFO fields will be processed together. If set to `False`, each INFO field will
1789        be processed individually. The default value is, defaults to False
1790        :type proccess_all_fields_together: bool (optional)
1791        :param table: The `table` parameter in the `explode_infos` function is used to specify the name
1792        of the table where the exploded INFO fields will be added as individual columns. If you provide
1793        a value for the `table` parameter, the function will use that table name. If the `table`
1794        parameter is
1795        :type table: str
1796        :return: The `explode_infos` function returns a list of added columns.
1797        """
1798
1799        # drop indexes
1800        self.drop_indexes()
1801
1802        # connexion format
1803        connexion_format = self.get_connexion_format()
1804
1805        # Access
1806        access = self.get_config().get("access", None)
1807
1808        # Added columns
1809        added_columns = []
1810
1811        if access not in ["RO"]:
1812
1813            # prefix
1814            if prefix in [None, True] or not isinstance(prefix, str):
1815                if self.get_explode_infos_prefix() not in [None, True]:
1816                    prefix = self.get_explode_infos_prefix()
1817                else:
1818                    prefix = "INFO/"
1819
1820            # table variants
1821            if table is not None:
1822                table_variants = table
1823            else:
1824                table_variants = self.get_table_variants(clause="select")
1825
1826            # extra infos
1827            try:
1828                extra_infos = self.get_extra_infos()
1829            except:
1830                extra_infos = []
1831
1832            # Header infos
1833            header_infos = self.get_header().infos
1834
1835            log.debug(
1836                f"Explode INFO fields - ADD [{len(header_infos)}] annotations fields"
1837            )
1838
1839            sql_info_alter_table_array = []
1840
1841            # Info fields to check
1842            fields_list = list(header_infos)
1843            if fields:
1844                fields_list += fields
1845            fields_list = set(fields_list)
1846
1847            # If no fields
1848            if not fields:
1849                fields = []
1850
1851            # Translate fields if patterns
1852            fields = self.get_explode_infos_fields(explode_infos_fields=fields)
1853
1854            for info in fields:
1855
1856                info_id_sql = prefix + info
1857
1858                if (
1859                    info in fields_list
1860                    or prefix + info in fields_list
1861                    or info in extra_infos
1862                ):
1863
1864                    log.debug(f"Explode INFO fields - ADD '{info}' annotations fields")
1865
1866                    if info in header_infos:
1867                        info_type = header_infos[info].type
1868                        info_num = header_infos[info].num
1869                    else:
1870                        info_type = "String"
1871                        info_num = 0
1872
1873                    type_sql = self.code_type_map_to_sql.get(info_type, "VARCHAR")
1874                    if info_num != 1:
1875                        type_sql = "VARCHAR"
1876
1877                    # Add field
1878                    added_column = self.add_column(
1879                        table_name=table_variants,
1880                        column_name=info_id_sql,
1881                        column_type=type_sql,
1882                        default_value="null",
1883                        drop=force,
1884                    )
1885
1886                    if added_column:
1887                        added_columns.append(added_column)
1888
1889                    if added_column or force:
1890
1891                        # add field to index
1892                        self.index_additionnal_fields.append(info_id_sql)
1893
1894                        # Update field array
1895                        if connexion_format in ["duckdb"]:
1896                            update_info_field = f"""
1897                            "{info_id_sql}" =
1898                                CASE
1899                                    WHEN REGEXP_EXTRACT(concat(';', INFO), ';{info}=([^;]*)',1) IN ('','.') THEN NULL
1900                                    ELSE REGEXP_EXTRACT(concat(';', INFO), ';{info}=([^;]*)',1)
1901                                END
1902                            """
1903                        elif connexion_format in ["sqlite"]:
1904                            update_info_field = f"""
1905                                "{info_id_sql}" =
1906                                    CASE
1907                                        WHEN instr(INFO, '{info}=') = 0 THEN NULL
1908                                        WHEN instr(substr(INFO, instr(INFO, '{info}=')+{len(info)+1}),';') = 0 THEN substr(substr(INFO, instr(INFO, '{info}=')+{len(info)+1}), instr(substr(INFO, instr(INFO, '{info}=')+{len(info)+1}), '=')+1)
1909                                        ELSE substr(substr(INFO, instr(INFO, '{info}=')+{len(info)+1}), instr(substr(INFO, instr(INFO, '{info}=')+{len(info)+1}), '=')+1, instr(substr(INFO, instr(INFO, '{info}=')+{len(info)+1}),';')-instr(substr(INFO, instr(INFO, '{info}=')+{len(info)+1}), '=')-1)
1910                                    END
1911                            """
1912
1913                        sql_info_alter_table_array.append(update_info_field)
1914
1915            if sql_info_alter_table_array:
1916
1917                # By chromosomes
1918                try:
1919                    chromosomes_list = list(
1920                        self.get_query_to_df(
1921                            f""" SELECT "#CHROM" FROM {table_variants} GROUP BY "#CHROM" """
1922                        )["#CHROM"]
1923                    )
1924                except:
1925                    chromosomes_list = [None]
1926
1927                for chrom in chromosomes_list:
1928                    log.debug(f"Explode INFO fields - Chromosome {chrom}...")
1929
1930                    # Where clause
1931                    where_clause = ""
1932                    if chrom and len(chromosomes_list) > 1:
1933                        where_clause = f""" WHERE "#CHROM" = '{chrom}' """
1934
1935                    # Update table
1936                    if proccess_all_fields_together:
1937                        sql_info_alter_table_array_join = ", ".join(
1938                            sql_info_alter_table_array
1939                        )
1940                        if sql_info_alter_table_array_join:
1941                            sql_info_alter_table = f"""
1942                                UPDATE {table_variants}
1943                                SET {sql_info_alter_table_array_join}
1944                                {where_clause}
1945                                """
1946                            log.debug(
1947                                f"Explode INFO fields - Explode all {len(sql_info_alter_table_array)} fields..."
1948                            )
1949                            # log.debug(sql_info_alter_table)
1950                            self.conn.execute(sql_info_alter_table)
1951                    else:
1952                        sql_info_alter_num = 0
1953                        for sql_info_alter in sql_info_alter_table_array:
1954                            sql_info_alter_num += 1
1955                            sql_info_alter_table = f"""
1956                                UPDATE {table_variants}
1957                                SET {sql_info_alter}
1958                                {where_clause}
1959                                """
1960                            log.debug(
1961                                f"Explode INFO fields - Explode field {sql_info_alter_num}/{len(sql_info_alter_table_array)}..."
1962                            )
1963                            # log.debug(sql_info_alter_table)
1964                            self.conn.execute(sql_info_alter_table)
1965
1966        # create indexes
1967        if create_index:
1968            self.create_indexes()
1969
1970        return added_columns

The explode_infos function in Python takes a VCF file and explodes the INFO fields into individual columns, returning a list of added columns.

Parameters

prefix: The prefix parameter is a string that is used as a prefix for the exploded INFO fields. If the prefix is not provided or is set to None, the function will use the value of self.get_explode_infos_prefix() as the prefix
create_index: The create_index parameter is a boolean flag that specifies whether to create indexes on the exploded INFO fields. If set to True, indexes will be created; if set to False, indexes will not be created. The default value is False, defaults to False
fields: The fields parameter in the explode_infos function is a list of INFO fields that you want to explode into individual columns. If this parameter is not provided, all INFO fields will be exploded. You can specify the INFO fields you want to explode by passing them as a list to the `
force: The force parameter in the explode_infos function is a boolean flag that determines whether to drop and recreate a column if it already exists in the table. If force is set to True, the column will be dropped and recreated. If force is set to `False, defaults to False
proccess_all_fields_together: The proccess_all_fields_together parameter is a boolean flag that determines whether to process all the INFO fields together or individually. If set to True, all the INFO fields will be processed together. If set to False, each INFO field will be processed individually. The default value is, defaults to False
table: The table parameter in the explode_infos function is used to specify the name of the table where the exploded INFO fields will be added as individual columns. If you provide a value for the table parameter, the function will use that table name. If the table parameter is

Returns

The explode_infos function returns a list of added columns.

def create_indexes(self) -> None: View Source

1972    def create_indexes(self) -> None:
1973        """
1974        Create indexes on the table after insertion
1975        """
1976
1977        # Access
1978        access = self.get_config().get("access", None)
1979
1980        # get table variants
1981        table_variants = self.get_table_variants("FROM")
1982
1983        if self.get_indexing() and access not in ["RO"]:
1984            # Create index
1985            sql_create_table_index = f'CREATE INDEX IF NOT EXISTS idx_{self.get_table_variants()} ON {table_variants} ("#CHROM", "POS", "REF", "ALT")'
1986            self.conn.execute(sql_create_table_index)
1987            sql_create_table_index = f'CREATE INDEX IF NOT EXISTS idx_{self.get_table_variants()}_chrom ON {table_variants} ("#CHROM")'
1988            self.conn.execute(sql_create_table_index)
1989            sql_create_table_index = f'CREATE INDEX IF NOT EXISTS idx_{self.get_table_variants()}_pos ON {table_variants} ("POS")'
1990            self.conn.execute(sql_create_table_index)
1991            sql_create_table_index = f'CREATE INDEX IF NOT EXISTS idx_{self.get_table_variants()}_ref ON {table_variants} ( "REF")'
1992            self.conn.execute(sql_create_table_index)
1993            sql_create_table_index = f'CREATE INDEX IF NOT EXISTS idx_{self.get_table_variants()}_alt ON {table_variants} ("ALT")'
1994            self.conn.execute(sql_create_table_index)
1995            for field in self.index_additionnal_fields:
1996                sql_create_table_index = f""" CREATE INDEX IF NOT EXISTS "idx_{self.get_table_variants()}_{field}" ON {table_variants} ("{field}") """
1997                self.conn.execute(sql_create_table_index)

Create indexes on the table after insertion

def drop_indexes(self) -> None: View Source

1999    def drop_indexes(self) -> None:
2000        """
2001        Create indexes on the table after insertion
2002        """
2003
2004        # Access
2005        access = self.get_config().get("access", None)
2006
2007        # get table variants
2008        table_variants = self.get_table_variants("FROM")
2009
2010        # Get database format
2011        connexion_format = self.get_connexion_format()
2012
2013        if access not in ["RO"]:
2014            if connexion_format in ["duckdb"]:
2015                sql_list_indexes = f"SELECT index_name FROM duckdb_indexes WHERE table_name='{table_variants}'"
2016            elif connexion_format in ["sqlite"]:
2017                sql_list_indexes = f"SELECT name FROM sqlite_master WHERE type='index' AND tbl_name='{table_variants}';"
2018
2019            list_indexes = self.conn.execute(sql_list_indexes)
2020            index_names = [row[0] for row in list_indexes.fetchall()]
2021            for index in index_names:
2022                sql_drop_table_index = f""" DROP INDEX IF EXISTS "{index}" """
2023                self.conn.execute(sql_drop_table_index)

Create indexes on the table after insertion

def read_vcf_header(self, f) -> list: View Source

2025    def read_vcf_header(self, f) -> list:
2026        """
2027        It reads the header of a VCF file and returns a list of the header lines
2028
2029        :param f: the file object
2030        :return: The header lines of the VCF file.
2031        """
2032
2033        header_list = []
2034        for line in f:
2035            header_list.append(line)
2036            if line.startswith("#CHROM"):
2037                break
2038        return header_list

It reads the header of a VCF file and returns a list of the header lines

Parameters

f: the file object

Returns

The header lines of the VCF file.

def read_vcf_header_file(self, file: str = None) -> list: View Source

2040    def read_vcf_header_file(self, file: str = None) -> list:
2041        """
2042        The `read_vcf_header_file` function reads the header of a VCF file, handling both compressed and
2043        uncompressed files.
2044
2045        :param file: The `file` parameter is a string that represents the path to the VCF header file
2046        that you want to read. It is an optional parameter, so if you don't provide a value, it will
2047        default to `None`
2048        :type file: str
2049        :return: The function `read_vcf_header_file` returns a list.
2050        """
2051
2052        if self.get_input_compressed(input_file=file):
2053            with bgzf.open(file, "rt") as f:
2054                return self.read_vcf_header(f=f)
2055        else:
2056            with open(file, "rt") as f:
2057                return self.read_vcf_header(f=f)

The read_vcf_header_file function reads the header of a VCF file, handling both compressed and uncompressed files.

Parameters

file: The file parameter is a string that represents the path to the VCF header file that you want to read. It is an optional parameter, so if you don't provide a value, it will default to None

Returns

The function read_vcf_header_file returns a list.

def execute_query(self, query: str): View Source

2059    def execute_query(self, query: str):
2060        """
2061        It takes a query as an argument, executes it, and returns the results
2062
2063        :param query: The query to be executed
2064        :return: The result of the query is being returned.
2065        """
2066        if query:
2067            return self.conn.execute(query)  # .fetchall()
2068        else:
2069            return None

It takes a query as an argument, executes it, and returns the results

Parameters

query: The query to be executed

Returns

The result of the query is being returned.

2071    def export_output(
2072        self,
2073        output_file: str | None = None,
2074        output_header: str | None = None,
2075        export_header: bool = True,
2076        query: str | None = None,
2077        parquet_partitions: list | None = None,
2078        chunk_size: int | None = None,
2079        threads: int | None = None,
2080        sort: bool = False,
2081        index: bool = False,
2082        order_by: str | None = None,
2083    ) -> bool:
2084        """
2085        The `export_output` function exports data from a VCF file to a specified output file in various
2086        formats, including VCF, CSV, TSV, PSV, and Parquet.
2087
2088        :param output_file: The `output_file` parameter is a string that specifies the name of the
2089        output file to be generated by the function. This is where the exported data will be saved
2090        :type output_file: str
2091        :param output_header: The `output_header` parameter is a string that specifies the name of the
2092        file where the header of the VCF file will be exported. If this parameter is not provided, the
2093        header will be exported to a file with the same name as the `output_file` parameter, but with
2094        the extension "
2095        :type output_header: str
2096        :param export_header: The `export_header` parameter is a boolean flag that determines whether
2097        the header of a VCF file should be exported to a separate file or not. If `export_header` is
2098        True, the header will be exported to a file. If `export_header` is False, the header will not
2099        be, defaults to True, if output format is not VCF
2100        :type export_header: bool (optional)
2101        :param query: The `query` parameter is an optional SQL query that can be used to filter and
2102        select specific data from the VCF file before exporting it. If provided, only the data that
2103        matches the query will be exported
2104        :type query: str
2105        :param parquet_partitions: The `parquet_partitions` parameter is a list that specifies the
2106        columns to be used for partitioning the Parquet file during export. Partitioning is a way to
2107        organize data in a hierarchical directory structure based on the values of one or more columns.
2108        This can improve query performance when working with large datasets
2109        :type parquet_partitions: list
2110        :param chunk_size: The `chunk_size` parameter specifies the number of
2111        records in batch when exporting data in Parquet format. This parameter is used for
2112        partitioning the Parquet file into multiple files.
2113        :type chunk_size: int
2114        :param threads: The `threads` parameter is an optional parameter that specifies the number of
2115        threads to be used during the export process. It determines the level of parallelism and can
2116        improve the performance of the export operation. If not provided, the function will use the
2117        default number of threads
2118        :type threads: int
2119        :param sort: The `sort` parameter is a boolean flag that determines whether the output file
2120        should be sorted or not. If `sort` is set to `True`, the output file will be sorted based on the
2121        genomic coordinates of the variants. By default, the value of `sort` is `False`, defaults to
2122        False
2123        :type sort: bool (optional)
2124        :param index: The `index` parameter is a boolean flag that determines whether an index should be
2125        created on the output file. If `index` is True, an index will be created. If `index` is False,
2126        no index will be created. The default value is False, defaults to False
2127        :type index: bool (optional)
2128        :param order_by: The `order_by` parameter is a string that specifies the column(s) to use for
2129        sorting the output file. This parameter is only applicable when exporting data in VCF format
2130        :type order_by: str
2131        :return: a boolean value. It checks if the output file exists and returns True if it does, or
2132        None if it doesn't.
2133        """
2134
2135        # Log
2136        log.info("Exporting...")
2137
2138        # Full path
2139        output_file = full_path(output_file)
2140        output_header = full_path(output_header)
2141
2142        # Config
2143        config = self.get_config()
2144
2145        # Param
2146        param = self.get_param()
2147
2148        # Tmp files to remove
2149        tmp_to_remove = []
2150
2151        # If no output, get it
2152        if not output_file:
2153            output_file = self.get_output()
2154
2155        # If not threads
2156        if not threads:
2157            threads = self.get_threads()
2158
2159        # Auto header name with extension
2160        if export_header or output_header:
2161            if not output_header:
2162                output_header = f"{output_file}.hdr"
2163            # Export header
2164            self.export_header(output_file=output_file)
2165
2166        # Switch off export header if VCF output
2167        output_file_type = get_file_format(output_file)
2168        if output_file_type in ["vcf"]:
2169            export_header = False
2170            tmp_to_remove.append(output_header)
2171
2172        # Chunk size
2173        if not chunk_size:
2174            chunk_size = config.get("chunk_size", None)
2175
2176        # Parquet partition
2177        if not parquet_partitions:
2178            parquet_partitions = param.get("export", {}).get("parquet_partitions", None)
2179        if parquet_partitions and isinstance(parquet_partitions, str):
2180            parquet_partitions = parquet_partitions.split(",")
2181
2182        # Order by
2183        if not order_by:
2184            order_by = param.get("export", {}).get("order_by", "")
2185
2186        # Header in output
2187        header_in_output = param.get("export", {}).get("include_header", False)
2188
2189        # Database
2190        database_source = self.get_connexion()
2191
2192        # Connexion format
2193        connexion_format = self.get_connexion_format()
2194
2195        # Explode infos
2196        if self.get_explode_infos():
2197            self.explode_infos(
2198                prefix=self.get_explode_infos_prefix(),
2199                fields=self.get_explode_infos_fields(),
2200                force=False,
2201            )
2202
2203        # if connexion_format in ["sqlite"] or query:
2204        if connexion_format in ["sqlite"]:
2205
2206            # Export in Parquet
2207            random_tmp = "".join(
2208                random.choice(string.ascii_lowercase) for i in range(10)
2209            )
2210            database_source = f"""{output_file}.{random_tmp}.database_export.parquet"""
2211            tmp_to_remove.append(database_source)
2212
2213            # Table Variants
2214            table_variants = self.get_table_variants()
2215
2216            # Create export query
2217            sql_query_export_subquery = f"""
2218                SELECT * FROM {table_variants}
2219                """
2220
2221            # Write source file
2222            fp.write(database_source, self.get_query_to_df(sql_query_export_subquery))
2223
2224        # Create database
2225        database = Database(
2226            database=database_source,
2227            table="variants",
2228            header_file=output_header,
2229            conn_config=self.get_connexion_config(),
2230        )
2231
2232        # Existing colomns header
2233        existing_columns_header = database.get_header_columns_from_database()
2234
2235        # Sample list
2236        get_samples = self.get_samples()
2237        get_samples_check = self.get_samples_check()
2238        samples_force = get_samples is not None
2239        sample_list = self.get_header_sample_list(
2240            check=get_samples_check, samples=get_samples, samples_force=samples_force
2241        )
2242
2243        # Export file
2244        database.export(
2245            output_database=output_file,
2246            output_header=output_header,
2247            existing_columns_header=existing_columns_header,
2248            parquet_partitions=parquet_partitions,
2249            chunk_size=chunk_size,
2250            threads=threads,
2251            sort=sort,
2252            index=index,
2253            header_in_output=header_in_output,
2254            order_by=order_by,
2255            query=query,
2256            export_header=export_header,
2257            sample_list=sample_list,
2258        )
2259
2260        # Remove
2261        remove_if_exists(tmp_to_remove)
2262
2263        return (os.path.exists(output_file) or None) and (
2264            os.path.exists(output_file) or None
2265        )

The export_output function exports data from a VCF file to a specified output file in various formats, including VCF, CSV, TSV, PSV, and Parquet.

Parameters

output_file: The output_file parameter is a string that specifies the name of the output file to be generated by the function. This is where the exported data will be saved
output_header: The output_header parameter is a string that specifies the name of the file where the header of the VCF file will be exported. If this parameter is not provided, the header will be exported to a file with the same name as the output_file parameter, but with the extension "
export_header: The export_header parameter is a boolean flag that determines whether the header of a VCF file should be exported to a separate file or not. If export_header is True, the header will be exported to a file. If export_header is False, the header will not be, defaults to True, if output format is not VCF
query: The query parameter is an optional SQL query that can be used to filter and select specific data from the VCF file before exporting it. If provided, only the data that matches the query will be exported
parquet_partitions: The parquet_partitions parameter is a list that specifies the columns to be used for partitioning the Parquet file during export. Partitioning is a way to organize data in a hierarchical directory structure based on the values of one or more columns. This can improve query performance when working with large datasets
chunk_size: The chunk_size parameter specifies the number of records in batch when exporting data in Parquet format. This parameter is used for partitioning the Parquet file into multiple files.
threads: The threads parameter is an optional parameter that specifies the number of threads to be used during the export process. It determines the level of parallelism and can improve the performance of the export operation. If not provided, the function will use the default number of threads
sort: The sort parameter is a boolean flag that determines whether the output file should be sorted or not. If sort is set to True, the output file will be sorted based on the genomic coordinates of the variants. By default, the value of sort is False, defaults to False
index: The index parameter is a boolean flag that determines whether an index should be created on the output file. If index is True, an index will be created. If index is False, no index will be created. The default value is False, defaults to False
order_by: The order_by parameter is a string that specifies the column(s) to use for sorting the output file. This parameter is only applicable when exporting data in VCF format

Returns

a boolean value. It checks if the output file exists and returns True if it does, or None if it doesn't.

def get_extra_infos(self, table: str = None) -> list: View Source

2267    def get_extra_infos(self, table: str = None) -> list:
2268        """
2269        The `get_extra_infos` function returns a list of columns that are in a specified table but not
2270        in the header.
2271
2272        :param table: The `table` parameter in the `get_extra_infos` function is used to specify the
2273        name of the table from which you want to retrieve the extra columns that are not present in the
2274        header. If the `table` parameter is not provided when calling the function, it will default to
2275        using the variants
2276        :type table: str
2277        :return: A list of columns that are in the specified table but not in the header of the table.
2278        """
2279
2280        header_columns = []
2281
2282        if not table:
2283            table = self.get_table_variants(clause="from")
2284            header_columns = self.get_header_columns()
2285
2286        # Check all columns in the database
2287        query = f""" SELECT * FROM {table} LIMIT 1 """
2288        log.debug(f"query {query}")
2289        table_columns = self.get_query_to_df(query).columns.tolist()
2290        extra_columns = []
2291
2292        # Construct extra infos (not in header)
2293        for column in table_columns:
2294            if column not in header_columns:
2295                extra_columns.append(column)
2296
2297        return extra_columns

The get_extra_infos function returns a list of columns that are in a specified table but not in the header.

Parameters

table: The table parameter in the get_extra_infos function is used to specify the name of the table from which you want to retrieve the extra columns that are not present in the header. If the table parameter is not provided when calling the function, it will default to using the variants

Returns

A list of columns that are in the specified table but not in the header of the table.

def get_extra_infos_sql(self, table: str = None) -> str: View Source

2299    def get_extra_infos_sql(self, table: str = None) -> str:
2300        """
2301        It returns a string of the extra infos, separated by commas, and each extra info is surrounded
2302        by double quotes
2303
2304        :param table: The name of the table to get the extra infos from. If None, the default table is
2305        used
2306        :type table: str
2307        :return: A string of the extra infos
2308        """
2309
2310        return ", ".join(
2311            ['"' + str(elem) + '"' for elem in self.get_extra_infos(table=table)]
2312        )

It returns a string of the extra infos, separated by commas, and each extra info is surrounded by double quotes

Parameters

table: The name of the table to get the extra infos from. If None, the default table is used

Returns

A string of the extra infos

def export_header( self, header_name: str = None, output_file: str = None, output_file_ext: str = '.hdr', clean_header: bool = True, remove_chrom_line: bool = False) -> str: View Source

2314    def export_header(
2315        self,
2316        header_name: str = None,
2317        output_file: str = None,
2318        output_file_ext: str = ".hdr",
2319        clean_header: bool = True,
2320        remove_chrom_line: bool = False,
2321    ) -> str:
2322        """
2323        The `export_header` function takes a VCF file, extracts the header, modifies it according to
2324        specified options, and writes it to a new file.
2325
2326        :param header_name: The `header_name` parameter is the name of the header file to be created. If
2327        this parameter is not specified, the header will be written to the output file
2328        :type header_name: str
2329        :param output_file: The `output_file` parameter in the `export_header` function is used to
2330        specify the name of the output file where the header will be written. If this parameter is not
2331        provided, the header will be written to a temporary file
2332        :type output_file: str
2333        :param output_file_ext: The `output_file_ext` parameter in the `export_header` function is a
2334        string that represents the extension of the output header file. By default, it is set to ".hdr"
2335        if not specified by the user. This extension will be appended to the `output_file` name to
2336        create the final, defaults to .hdr
2337        :type output_file_ext: str (optional)
2338        :param clean_header: The `clean_header` parameter in the `export_header` function is a boolean
2339        flag that determines whether the header should be cleaned or not. When `clean_header` is set to
2340        `True`, the function will clean the header by modifying certain lines based on a specific
2341        pattern. If `clean_header`, defaults to True
2342        :type clean_header: bool (optional)
2343        :param remove_chrom_line: The `remove_chrom_line` parameter in the `export_header` function is a
2344        boolean flag that determines whether the #CHROM line should be removed from the header before
2345        writing it to the output file. If set to `True`, the #CHROM line will be removed; if set to `,
2346        defaults to False
2347        :type remove_chrom_line: bool (optional)
2348        :return: The function `export_header` returns the name of the temporary header file that is
2349        created.
2350        """
2351
2352        if not header_name and not output_file:
2353            output_file = self.get_output()
2354
2355        if self.get_header():
2356
2357            # Get header object
2358            header_obj = self.get_header()
2359
2360            # Create database
2361            db_for_header = Database(database=self.get_input())
2362
2363            # Get real columns in the file
2364            db_header_columns = db_for_header.get_columns()
2365
2366            with tempfile.TemporaryDirectory() as tmpdir:
2367
2368                # Write header file
2369                header_file_tmp = os.path.join(tmpdir, "header")
2370                f = open(header_file_tmp, "w")
2371                vcf.Writer(f, header_obj)
2372                f.close()
2373
2374                # Replace #CHROM line with rel columns
2375                header_list = db_for_header.read_header_file(
2376                    header_file=header_file_tmp
2377                )
2378                header_list[-1] = "\t".join(db_header_columns)
2379
2380                # Remove CHROM line
2381                if remove_chrom_line:
2382                    header_list.pop()
2383
2384                # Clean header
2385                if clean_header:
2386                    header_list_clean = []
2387                    for head in header_list:
2388                        # Clean head for malformed header
2389                        head_clean = head
2390                        head_clean = re.subn(
2391                            "##FORMAT=<ID=(.*),Number=(.*),Type=Flag",
2392                            r"##FORMAT=<ID=\1,Number=\2,Type=String",
2393                            head_clean,
2394                            2,
2395                        )[0]
2396                        # Write header
2397                        header_list_clean.append(head_clean)
2398                    header_list = header_list_clean
2399
2400            tmp_header_name = output_file + output_file_ext
2401
2402            f = open(tmp_header_name, "w")
2403            for line in header_list:
2404                f.write(line)
2405            f.close()
2406
2407        return tmp_header_name

The export_header function takes a VCF file, extracts the header, modifies it according to specified options, and writes it to a new file.

Parameters

header_name: The header_name parameter is the name of the header file to be created. If this parameter is not specified, the header will be written to the output file
output_file: The output_file parameter in the export_header function is used to specify the name of the output file where the header will be written. If this parameter is not provided, the header will be written to a temporary file
output_file_ext: The output_file_ext parameter in the export_header function is a string that represents the extension of the output header file. By default, it is set to ".hdr" if not specified by the user. This extension will be appended to the output_file name to create the final, defaults to .hdr
clean_header: The clean_header parameter in the export_header function is a boolean flag that determines whether the header should be cleaned or not. When clean_header is set to True, the function will clean the header by modifying certain lines based on a specific pattern. If clean_header, defaults to True
remove_chrom_line: The remove_chrom_line parameter in the export_header function is a boolean flag that determines whether the #CHROM line should be removed from the header before writing it to the output file. If set to True, the #CHROM line will be removed; if set to `, defaults to False

Returns

The function export_header returns the name of the temporary header file that is created.

def export_variant_vcf( self, vcf_file, remove_info: bool = False, add_samples: bool = True, list_samples: list = [], where_clause: str = '', index: bool = False, threads: int | None = None) -> bool | None: View Source

2409    def export_variant_vcf(
2410        self,
2411        vcf_file,
2412        remove_info: bool = False,
2413        add_samples: bool = True,
2414        list_samples: list = [],
2415        where_clause: str = "",
2416        index: bool = False,
2417        threads: int | None = None,
2418    ) -> bool | None:
2419        """
2420        The `export_variant_vcf` function exports a VCF file with specified samples, allowing options to
2421        remove INFO field, add samples, and control compression and indexing.
2422
2423        :param vcf_file: The `vcf_file` parameter is the name of the file where the VCF data will be
2424        written to. It is the output file that will contain the filtered VCF data based on the specified
2425        parameters
2426        :param remove_info: The `remove_info` parameter in the `export_variant_vcf` function is a
2427        boolean flag that determines whether to remove the INFO field from the output VCF file. If set
2428        to `True`, the INFO field will be removed. If set to `False`, the INFO field will be included
2429        in, defaults to False
2430        :type remove_info: bool (optional)
2431        :param add_samples: The `add_samples` parameter is a boolean parameter that determines whether
2432        the samples should be added to the VCF file or not. If set to True, the samples will be added.
2433        If set to False, the samples will be removed. The default value is True, defaults to True
2434        :type add_samples: bool (optional)
2435        :param list_samples: The `list_samples` parameter is a list of samples that you want to include
2436        in the output VCF file. By default, all samples will be included. If you provide a list of
2437        samples, only those samples will be included in the output file
2438        :type list_samples: list
2439        :param index: The `index` parameter in the `export_variant_vcf` function is a boolean flag that
2440        determines whether or not to create an index for the output VCF file. If `index` is set to
2441        `True`, the output VCF file will be indexed using tabix. If `index`, defaults to False
2442        :type index: bool (optional)
2443        :param threads: The `threads` parameter in the `export_variant_vcf` function specifies the
2444        number of threads to use for exporting the VCF file. It determines how many parallel threads
2445        will be used during the export process. More threads can potentially speed up the export process
2446        by utilizing multiple cores of the processor. If
2447        :type threads: int | None
2448        :return: The `export_variant_vcf` function returns the result of calling the `export_output`
2449        method with various parameters including the output file, query, threads, sort flag, and index
2450        flag. The `export_output` method is responsible for exporting the VCF data based on the
2451        specified parameters and configurations provided in the `export_variant_vcf` function.
2452        """
2453
2454        # Config
2455        config = self.get_config()
2456
2457        # Extract VCF
2458        log.debug("Export VCF...")
2459
2460        # Table variants
2461        table_variants = self.get_table_variants()
2462
2463        # Threads
2464        if not threads:
2465            threads = self.get_threads()
2466
2467        # Info fields
2468        if remove_info:
2469            if not isinstance(remove_info, str):
2470                remove_info = "."
2471            info_field = f"""'{remove_info}' as INFO"""
2472        else:
2473            info_field = "INFO"
2474
2475        # Samples fields
2476        if add_samples:
2477            if not list_samples:
2478                list_samples = self.get_header_sample_list()
2479            if list_samples:
2480                samples_fields = " , FORMAT , " + " , ".join(list_samples)
2481            else:
2482                samples_fields = ""
2483            log.debug(f"samples_fields: {samples_fields}")
2484        else:
2485            samples_fields = ""
2486
2487        # Where clause
2488        if where_clause is None:
2489            where_clause = ""
2490
2491        # Variants
2492        select_fields = """ "#CHROM", POS, ID, REF, ALT, QUAL, FILTER """
2493        sql_query_select = f""" SELECT {select_fields}, {info_field} {samples_fields} FROM {table_variants} {where_clause} """
2494        log.debug(f"sql_query_select={sql_query_select}")
2495
2496        return self.export_output(
2497            output_file=vcf_file,
2498            output_header=None,
2499            export_header=True,
2500            query=sql_query_select,
2501            parquet_partitions=None,
2502            chunk_size=config.get("chunk_size", None),
2503            threads=threads,
2504            sort=True,
2505            index=index,
2506            order_by=None,
2507        )

The export_variant_vcf function exports a VCF file with specified samples, allowing options to remove INFO field, add samples, and control compression and indexing.

Parameters

vcf_file: The vcf_file parameter is the name of the file where the VCF data will be written to. It is the output file that will contain the filtered VCF data based on the specified parameters
remove_info: The remove_info parameter in the export_variant_vcf function is a boolean flag that determines whether to remove the INFO field from the output VCF file. If set to True, the INFO field will be removed. If set to False, the INFO field will be included in, defaults to False
add_samples: The add_samples parameter is a boolean parameter that determines whether the samples should be added to the VCF file or not. If set to True, the samples will be added. If set to False, the samples will be removed. The default value is True, defaults to True
list_samples: The list_samples parameter is a list of samples that you want to include in the output VCF file. By default, all samples will be included. If you provide a list of samples, only those samples will be included in the output file
index: The index parameter in the export_variant_vcf function is a boolean flag that determines whether or not to create an index for the output VCF file. If index is set to True, the output VCF file will be indexed using tabix. If index, defaults to False
threads: The threads parameter in the export_variant_vcf function specifies the number of threads to use for exporting the VCF file. It determines how many parallel threads will be used during the export process. More threads can potentially speed up the export process by utilizing multiple cores of the processor. If

Returns

The export_variant_vcf function returns the result of calling the export_output method with various parameters including the output file, query, threads, sort flag, and index flag. The export_output method is responsible for exporting the VCF data based on the specified parameters and configurations provided in the export_variant_vcf function.

def run_commands(self, commands: list = [], threads: int = 1) -> None: View Source

2509    def run_commands(self, commands: list = [], threads: int = 1) -> None:
2510        """
2511        It takes a list of commands and runs them in parallel using the number of threads specified
2512
2513        :param commands: A list of commands to run
2514        :param threads: The number of threads to use, defaults to 1 (optional)
2515        """
2516
2517        run_parallel_commands(commands, threads)

It takes a list of commands and runs them in parallel using the number of threads specified

Parameters

commands: A list of commands to run
threads: The number of threads to use, defaults to 1 (optional)

def get_threads(self, default: int = 1) -> int: View Source

2519    def get_threads(self, default: int = 1) -> int:
2520        """
2521        This function returns the number of threads to use for a job, with a default value of 1 if not
2522        specified.
2523
2524        :param default: The `default` parameter in the `get_threads` method is used to specify the
2525        default number of threads to use if no specific value is provided. If no value is provided for
2526        the `threads` parameter in the configuration or input parameters, the `default` value will be
2527        used, defaults to 1
2528        :type default: int (optional)
2529        :return: the number of threads to use for the current job.
2530        """
2531
2532        # Config
2533        config = self.get_config()
2534
2535        # Param
2536        param = self.get_param()
2537
2538        # Input threads
2539        input_thread = param.get("threads", config.get("threads", None))
2540
2541        # Check threads
2542        if not input_thread:
2543            threads = default
2544        elif int(input_thread) <= 0:
2545            threads = os.cpu_count()
2546        else:
2547            threads = int(input_thread)
2548        return threads

This function returns the number of threads to use for a job, with a default value of 1 if not specified.

Parameters

default: The default parameter in the get_threads method is used to specify the default number of threads to use if no specific value is provided. If no value is provided for the threads parameter in the configuration or input parameters, the default value will be used, defaults to 1

Returns

the number of threads to use for the current job.

def get_memory(self, default: str = None) -> str: View Source

2550    def get_memory(self, default: str = None) -> str:
2551        """
2552        This function retrieves the memory value from parameters or configuration with a default value
2553        if not found.
2554
2555        :param default: The `get_memory` function takes in a default value as a string parameter. This
2556        default value is used as a fallback in case the `memory` parameter is not provided in the
2557        `param` dictionary or the `config` dictionary. If `memory` is not found in either dictionary,
2558        the function
2559        :type default: str
2560        :return: The `get_memory` function returns a string value representing the memory parameter. If
2561        the `input_memory` is provided in the parameters, it will return that value. Otherwise, it will
2562        return the default value provided as an argument to the function.
2563        """
2564
2565        # Config
2566        config = self.get_config()
2567
2568        # Param
2569        param = self.get_param()
2570
2571        # Input threads
2572        input_memory = param.get("memory", config.get("memory", None))
2573
2574        # Check threads
2575        if input_memory:
2576            memory = input_memory
2577        else:
2578            memory = default
2579
2580        return memory

This function retrieves the memory value from parameters or configuration with a default value if not found.

Parameters

default: The get_memory function takes in a default value as a string parameter. This default value is used as a fallback in case the memory parameter is not provided in the param dictionary or the config dictionary. If memory is not found in either dictionary, the function

Returns

The get_memory function returns a string value representing the memory parameter. If the input_memory is provided in the parameters, it will return that value. Otherwise, it will return the default value provided as an argument to the function.

def update_from_vcf(self, vcf_file: str) -> None: View Source

2582    def update_from_vcf(self, vcf_file: str) -> None:
2583        """
2584        > If the database is duckdb, then use the parquet method, otherwise use the sqlite method
2585
2586        :param vcf_file: the path to the VCF file
2587        """
2588
2589        connexion_format = self.get_connexion_format()
2590
2591        if connexion_format in ["duckdb"]:
2592            self.update_from_vcf_duckdb(vcf_file)
2593        elif connexion_format in ["sqlite"]:
2594            self.update_from_vcf_sqlite(vcf_file)

If the database is duckdb, then use the parquet method, otherwise use the sqlite method

Parameters

vcf_file: the path to the VCF file

def update_from_vcf_duckdb(self, vcf_file: str) -> None: View Source

2596    def update_from_vcf_duckdb(self, vcf_file: str) -> None:
2597        """
2598        It takes a VCF file and updates the INFO column of the variants table in the database with the
2599        INFO column of the VCF file
2600
2601        :param vcf_file: the path to the VCF file
2602        """
2603
2604        # varaints table
2605        table_variants = self.get_table_variants()
2606
2607        # Loading VCF into temporaire table
2608        skip = self.get_header_length(file=vcf_file)
2609        vcf_df = pd.read_csv(
2610            vcf_file,
2611            sep="\t",
2612            engine="c",
2613            skiprows=skip,
2614            header=0,
2615            low_memory=False,
2616        )
2617        sql_query_update = f"""
2618        UPDATE {table_variants} as table_variants
2619            SET INFO = concat(
2620                            CASE
2621                                WHEN INFO NOT IN ('', '.')
2622                                THEN INFO
2623                                ELSE ''
2624                            END,
2625                            (
2626                                SELECT 
2627                                    concat(
2628                                        CASE
2629                                            WHEN table_variants.INFO NOT IN ('','.') AND table_parquet.INFO NOT IN ('','.')
2630                                            THEN ';'
2631                                            ELSE ''
2632                                        END
2633                                        ,
2634                                        CASE
2635                                            WHEN table_parquet.INFO NOT IN ('','.')
2636                                            THEN table_parquet.INFO
2637                                            ELSE ''
2638                                        END
2639                                    )
2640                                FROM vcf_df as table_parquet
2641                                        WHERE CAST(table_parquet.\"#CHROM\" AS VARCHAR) = CAST(table_variants.\"#CHROM\" AS VARCHAR)
2642                                        AND table_parquet.\"POS\" = table_variants.\"POS\"
2643                                        AND table_parquet.\"ALT\" = table_variants.\"ALT\"
2644                                        AND table_parquet.\"REF\" = table_variants.\"REF\"
2645                                        AND table_parquet.INFO NOT IN ('','.')
2646                            )
2647                        )
2648            ;
2649            """
2650        self.conn.execute(sql_query_update)

It takes a VCF file and updates the INFO column of the variants table in the database with the INFO column of the VCF file

Parameters

vcf_file: the path to the VCF file

def update_from_vcf_sqlite(self, vcf_file: str) -> None: View Source

2652    def update_from_vcf_sqlite(self, vcf_file: str) -> None:
2653        """
2654        It creates a temporary table in the SQLite database, loads the VCF file into the temporary
2655        table, then updates the INFO column of the variants table with the INFO column of the temporary
2656        table
2657
2658        :param vcf_file: The path to the VCF file you want to update the database with
2659        """
2660
2661        # Create a temporary table for the VCF
2662        table_vcf = "tmp_vcf"
2663        sql_create = (
2664            f"CREATE TEMPORARY TABLE {table_vcf} AS SELECT * FROM variants WHERE 0"
2665        )
2666        self.conn.execute(sql_create)
2667
2668        # Loading VCF into temporaire table
2669        vcf_df = pd.read_csv(
2670            vcf_file, sep="\t", comment="#", header=None, low_memory=False
2671        )
2672        vcf_df.columns = ["#CHROM", "POS", "ID", "REF", "ALT", "QUAL", "FILTER", "INFO"]
2673        vcf_df.to_sql(table_vcf, self.conn, if_exists="append", index=False)
2674
2675        # Update table 'variants' with VCF data
2676        # warning: CONCAT as || operator
2677        sql_query_update = f"""
2678            UPDATE variants as table_variants
2679            SET INFO = CASE
2680                            WHEN INFO NOT IN ('', '.')
2681                            THEN INFO
2682                            ELSE ''
2683                        END ||
2684                        (
2685                        SELECT 
2686                            CASE 
2687                                WHEN table_variants.INFO NOT IN ('','.') 
2688                                    AND table_vcf.INFO NOT IN ('','.')  
2689                                THEN ';' 
2690                                ELSE '' 
2691                            END || 
2692                            CASE 
2693                                WHEN table_vcf.INFO NOT IN ('','.') 
2694                                THEN table_vcf.INFO 
2695                                ELSE '' 
2696                            END
2697                        FROM {table_vcf} as table_vcf
2698                        WHERE table_vcf.\"#CHROM\" = table_variants.\"#CHROM\"
2699                            AND table_vcf.\"POS\" = table_variants.\"POS\"
2700                            AND table_vcf.\"ALT\" = table_variants.\"ALT\"
2701                            AND table_vcf.\"REF\" = table_variants.\"REF\"
2702                        )
2703        """
2704        self.conn.execute(sql_query_update)
2705
2706        # Drop temporary table
2707        sql_drop = f"DROP TABLE {table_vcf}"
2708        self.conn.execute(sql_drop)

It creates a temporary table in the SQLite database, loads the VCF file into the temporary table, then updates the INFO column of the variants table with the INFO column of the temporary table

Parameters

vcf_file: The path to the VCF file you want to update the database with

def drop_variants_table(self) -> None: View Source

2710    def drop_variants_table(self) -> None:
2711        """
2712        > This function drops the variants table
2713        """
2714
2715        table_variants = self.get_table_variants()
2716        sql_table_variants = f"DROP TABLE IF EXISTS {table_variants}"
2717        self.conn.execute(sql_table_variants)

This function drops the variants table

def set_variant_id(self, variant_id_column: str = 'variant_id', force: bool = None) -> str: View Source

2719    def set_variant_id(
2720        self, variant_id_column: str = "variant_id", force: bool = None
2721    ) -> str:
2722        """
2723        It adds a column to the variants table called `variant_id` and populates it with a hash of the
2724        `#CHROM`, `POS`, `REF`, and `ALT` columns
2725
2726        :param variant_id_column: The name of the column to be created in the variants table, defaults
2727        to variant_id
2728        :type variant_id_column: str (optional)
2729        :param force: If True, the variant_id column will be created even if it already exists
2730        :type force: bool
2731        :return: The name of the column that contains the variant_id
2732        """
2733
2734        # Assembly
2735        assembly = self.get_param().get(
2736            "assembly", self.get_config().get("assembly", DEFAULT_ASSEMBLY)
2737        )
2738
2739        # INFO/Tag prefix
2740        prefix = self.get_explode_infos_prefix()
2741
2742        # Explode INFO/SVTYPE
2743        added_columns = self.explode_infos(prefix=prefix, fields=["SVTYPE"])
2744
2745        # variants table
2746        table_variants = self.get_table_variants()
2747
2748        # variant_id column
2749        if not variant_id_column:
2750            variant_id_column = "variant_id"
2751
2752        # Creta variant_id column
2753        if "variant_id" not in self.get_extra_infos() or force:
2754
2755            # Create column
2756            self.add_column(
2757                table_name=table_variants,
2758                column_name=variant_id_column,
2759                column_type="UBIGINT",
2760                default_value="0",
2761            )
2762
2763            # Update column
2764            self.conn.execute(
2765                f"""
2766                    UPDATE {table_variants}
2767                    SET "{variant_id_column}" = hash('{assembly}', "#CHROM", "POS", "REF", "ALT", '"{prefix}SVTYPE"')
2768                """
2769            )
2770
2771        # Remove added columns
2772        for added_column in added_columns:
2773            self.drop_column(column=added_column)
2774
2775        # return variant_id column name
2776        return variant_id_column

It adds a column to the variants table called variant_id and populates it with a hash of the #CHROM, POS, REF, and ALT columns

Parameters

variant_id_column: The name of the column to be created in the variants table, defaults to variant_id
force: If True, the variant_id column will be created even if it already exists

Returns

The name of the column that contains the variant_id

def get_variant_id_column(self, variant_id_column: str = 'variant_id', force: bool = None) -> str: View Source

2778    def get_variant_id_column(
2779        self, variant_id_column: str = "variant_id", force: bool = None
2780    ) -> str:
2781        """
2782        This function returns the variant_id column name
2783
2784        :param variant_id_column: The name of the column in the dataframe that contains the variant IDs,
2785        defaults to variant_id
2786        :type variant_id_column: str (optional)
2787        :param force: If True, will force the variant_id to be set to the value of variant_id_column. If
2788        False, will only set the variant_id if it is not already set. If None, will set the variant_id
2789        if it is not already set, or if it is set
2790        :type force: bool
2791        :return: The variant_id column name.
2792        """
2793
2794        return self.set_variant_id(variant_id_column=variant_id_column, force=force)

This function returns the variant_id column name

Parameters

variant_id_column: The name of the column in the dataframe that contains the variant IDs, defaults to variant_id
force: If True, will force the variant_id to be set to the value of variant_id_column. If False, will only set the variant_id if it is not already set. If None, will set the variant_id if it is not already set, or if it is set

Returns

The variant_id column name.

def scan_databases( self, database_formats: list = ['parquet'], database_releases: list = ['current']) -> dict: View Source

2800    def scan_databases(
2801        self,
2802        database_formats: list = ["parquet"],
2803        database_releases: list = ["current"],
2804    ) -> dict:
2805        """
2806        The function `scan_databases` scans for available databases based on specified formats and
2807        releases.
2808
2809        :param database_formats: The `database_formats` parameter is a list that specifies the formats
2810        of the databases to be scanned. In this case, the accepted format is "parquet"
2811        :type database_formats: list ["parquet"]
2812        :param database_releases: The `database_releases` parameter is a list that specifies the
2813        releases of the databases to be scanned. In the provided function, the default value for
2814        `database_releases` is set to `["current"]`, meaning that by default, the function will scan
2815        databases that are in the "current"
2816        :type database_releases: list
2817        :return: The function `scan_databases` returns a dictionary containing information about
2818        databases that match the specified formats and releases.
2819        """
2820
2821        # Config
2822        config = self.get_config()
2823
2824        # Param
2825        param = self.get_param()
2826
2827        # Param - Assembly
2828        assembly = param.get("assembly", config.get("assembly", None))
2829        if not assembly:
2830            assembly = DEFAULT_ASSEMBLY
2831            log.warning(f"Default assembly '{assembly}'")
2832
2833        # Scan for availabled databases
2834        log.info(
2835            f"Annotations - Check annotation parameters - Scan existing databases - Assembly {[assembly]} - Formats {database_formats} - Releases {database_releases}..."
2836        )
2837        databases_infos_dict = databases_infos(
2838            database_folder_releases=database_releases,
2839            database_formats=database_formats,
2840            assembly=assembly,
2841            config=config,
2842        )
2843        log.info(
2844            f"Annotations - Check annotation parameters - Scan existing databases - {len(databases_infos_dict)} databases found"
2845        )
2846
2847        return databases_infos_dict

The function scan_databases scans for available databases based on specified formats and releases.

Parameters

database_formats: The database_formats parameter is a list that specifies the formats of the databases to be scanned. In this case, the accepted format is "parquet"
database_releases: The database_releases parameter is a list that specifies the releases of the databases to be scanned. In the provided function, the default value for database_releases is set to ["current"], meaning that by default, the function will scan databases that are in the "current"

Returns

The function scan_databases returns a dictionary containing information about databases that match the specified formats and releases.

def annotation(self) -> None: View Source

2849    def annotation(self) -> None:
2850        """
2851        It annotates the VCF file with the annotations specified in the config file.
2852        """
2853
2854        # Config
2855        config = self.get_config()
2856
2857        # Param
2858        param = self.get_param()
2859
2860        # Param - Assembly
2861        assembly = param.get("assembly", config.get("assembly", None))
2862        if not assembly:
2863            assembly = DEFAULT_ASSEMBLY
2864            log.warning(f"Default assembly '{assembly}'")
2865
2866        # annotations databases folders
2867        annotations_databases = set(
2868            config.get("folders", {})
2869            .get("databases", {})
2870            .get("annotations", [DEFAULT_ANNOTATIONS_FOLDER])
2871            + config.get("folders", {})
2872            .get("databases", {})
2873            .get("parquet", ["~/howard/databases/parquet/current"])
2874            + config.get("folders", {})
2875            .get("databases", {})
2876            .get("bcftools", ["~/howard/databases/bcftools/current"])
2877        )
2878
2879        # Get param annotations
2880        if param.get("annotations", None) and isinstance(
2881            param.get("annotations", None), str
2882        ):
2883            log.debug(param.get("annotations", None))
2884            param_annotation_list = param.get("annotations").split(",")
2885        else:
2886            param_annotation_list = []
2887
2888        # Each tools param
2889        if param.get("annotation_parquet", None) != None:
2890            log.debug(
2891                f"""param.get("annotation_parquet", None)={param.get("annotation_parquet", None)}"""
2892            )
2893            if isinstance(param.get("annotation_parquet", None), list):
2894                param_annotation_list.append(",".join(param.get("annotation_parquet")))
2895            else:
2896                param_annotation_list.append(param.get("annotation_parquet"))
2897        if param.get("annotation_snpsift", None) != None:
2898            if isinstance(param.get("annotation_snpsift", None), list):
2899                param_annotation_list.append(
2900                    "snpsift:"
2901                    + "+".join(param.get("annotation_snpsift")).replace(",", "+")
2902                )
2903            else:
2904                param_annotation_list.append(
2905                    "snpsift:" + param.get("annotation_snpsift").replace(",", "+")
2906                )
2907        if param.get("annotation_snpeff", None) != None:
2908            param_annotation_list.append("snpeff:" + param.get("annotation_snpeff"))
2909        if param.get("annotation_bcftools", None) != None:
2910            if isinstance(param.get("annotation_bcftools", None), list):
2911                param_annotation_list.append(
2912                    "bcftools:"
2913                    + "+".join(param.get("annotation_bcftools")).replace(",", "+")
2914                )
2915            else:
2916                param_annotation_list.append(
2917                    "bcftools:" + param.get("annotation_bcftools").replace(",", "+")
2918                )
2919        if param.get("annotation_annovar", None) != None:
2920            param_annotation_list.append("annovar:" + param.get("annotation_annovar"))
2921        if param.get("annotation_exomiser", None) != None:
2922            param_annotation_list.append("exomiser:" + param.get("annotation_exomiser"))
2923        if param.get("annotation_splice", None) != None:
2924            param_annotation_list.append("splice:" + param.get("annotation_splice"))
2925
2926        # Merge param annotations list
2927        param["annotations"] = ",".join(param_annotation_list)
2928
2929        # debug
2930        log.debug(f"param_annotations={param['annotations']}")
2931
2932        if param.get("annotations"):
2933
2934            # Log
2935            # log.info("Annotations - Check annotation parameters")
2936
2937            if not "annotation" in param:
2938                param["annotation"] = {}
2939
2940            # List of annotations parameters
2941            annotations_list_input = {}
2942            if isinstance(param.get("annotations", None), str):
2943                annotation_file_list = [
2944                    value for value in param.get("annotations", "").split(",")
2945                ]
2946                for annotation_file in annotation_file_list:
2947                    annotations_list_input[annotation_file] = {"INFO": None}
2948            else:
2949                annotations_list_input = param.get("annotations", {})
2950
2951            log.info(f"Quick Annotations:")
2952            for annotation_key in list(annotations_list_input.keys()):
2953                log.info(f"   {annotation_key}")
2954
2955            # List of annotations and associated fields
2956            annotations_list = {}
2957
2958            for annotation_file in annotations_list_input:
2959
2960                # Explode annotations if ALL
2961                if (
2962                    annotation_file.upper() == "ALL"
2963                    or annotation_file.upper().startswith("ALL:")
2964                ):
2965
2966                    # check ALL parameters (formats, releases)
2967                    annotation_file_split = annotation_file.split(":")
2968                    database_formats = "parquet"
2969                    database_releases = "current"
2970                    for annotation_file_option in annotation_file_split[1:]:
2971                        database_all_options_split = annotation_file_option.split("=")
2972                        if database_all_options_split[0] == "format":
2973                            database_formats = database_all_options_split[1].split("+")
2974                        if database_all_options_split[0] == "release":
2975                            database_releases = database_all_options_split[1].split("+")
2976
2977                    # Scan for availabled databases
2978                    databases_infos_dict = self.scan_databases(
2979                        database_formats=database_formats,
2980                        database_releases=database_releases,
2981                    )
2982
2983                    # Add found databases in annotation parameters
2984                    for database_infos in databases_infos_dict.keys():
2985                        annotations_list[database_infos] = {"INFO": None}
2986
2987                else:
2988                    annotations_list[annotation_file] = annotations_list_input[
2989                        annotation_file
2990                    ]
2991
2992            # Check each databases
2993            if len(annotations_list):
2994
2995                log.info(
2996                    f"Annotations - Check annotation parameters - Check {len(annotations_list)} databases..."
2997                )
2998
2999                for annotation_file in annotations_list:
3000
3001                    # Init
3002                    annotations = annotations_list.get(annotation_file, None)
3003
3004                    # Annotation snpEff
3005                    if annotation_file.startswith("snpeff"):
3006
3007                        log.debug(f"Quick Annotation snpEff")
3008
3009                        if "snpeff" not in param["annotation"]:
3010                            param["annotation"]["snpeff"] = {}
3011
3012                        if "options" not in param["annotation"]["snpeff"]:
3013                            param["annotation"]["snpeff"]["options"] = ""
3014
3015                        # snpEff options in annotations
3016                        param["annotation"]["snpeff"]["options"] = "".join(
3017                            annotation_file.split(":")[1:]
3018                        )
3019
3020                    # Annotation Annovar
3021                    elif annotation_file.startswith("annovar"):
3022
3023                        log.debug(f"Quick Annotation Annovar")
3024
3025                        if "annovar" not in param["annotation"]:
3026                            param["annotation"]["annovar"] = {}
3027
3028                        if "annotations" not in param["annotation"]["annovar"]:
3029                            param["annotation"]["annovar"]["annotations"] = {}
3030
3031                        # Options
3032                        annotation_file_split = annotation_file.split(":")
3033                        for annotation_file_annotation in annotation_file_split[1:]:
3034                            if annotation_file_annotation:
3035                                param["annotation"]["annovar"]["annotations"][
3036                                    annotation_file_annotation
3037                                ] = annotations
3038
3039                    # Annotation Exomiser
3040                    elif annotation_file.startswith("exomiser"):
3041
3042                        log.debug(f"Quick Annotation Exomiser")
3043
3044                        param["annotation"]["exomiser"] = params_string_to_dict(
3045                            annotation_file
3046                        )
3047
3048                    # Annotation Splice
3049                    elif annotation_file.startswith("splice"):
3050
3051                        log.debug(f"Quick Annotation Splice")
3052
3053                        param["annotation"]["splice"] = params_string_to_dict(
3054                            annotation_file
3055                        )
3056
3057                    # Annotation Parquet or BCFTOOLS
3058                    else:
3059
3060                        # Tools detection
3061                        if annotation_file.startswith("bcftools:"):
3062                            annotation_tool_initial = "bcftools"
3063                            annotation_file = ":".join(annotation_file.split(":")[1:])
3064                        elif annotation_file.startswith("snpsift:"):
3065                            annotation_tool_initial = "snpsift"
3066                            annotation_file = ":".join(annotation_file.split(":")[1:])
3067                        else:
3068                            annotation_tool_initial = None
3069
3070                        # list of files
3071                        annotation_file_list = annotation_file.replace("+", ":").split(
3072                            ":"
3073                        )
3074
3075                        for annotation_file in annotation_file_list:
3076
3077                            if annotation_file:
3078
3079                                # Annotation tool initial
3080                                annotation_tool = annotation_tool_initial
3081
3082                                # Find file
3083                                annotation_file_found = None
3084
3085                                # Expand user
3086                                annotation_file = full_path(annotation_file)
3087
3088                                if os.path.exists(annotation_file):
3089                                    annotation_file_found = annotation_file
3090
3091                                else:
3092                                    # Find within assembly folders
3093                                    for annotations_database in annotations_databases:
3094                                        found_files = find_all(
3095                                            annotation_file,
3096                                            os.path.join(
3097                                                annotations_database, assembly
3098                                            ),
3099                                        )
3100                                        if len(found_files) > 0:
3101                                            annotation_file_found = found_files[0]
3102                                            break
3103                                    if not annotation_file_found and not assembly:
3104                                        # Find within folders
3105                                        for (
3106                                            annotations_database
3107                                        ) in annotations_databases:
3108                                            found_files = find_all(
3109                                                annotation_file, annotations_database
3110                                            )
3111                                            if len(found_files) > 0:
3112                                                annotation_file_found = found_files[0]
3113                                                break
3114                                log.debug(
3115                                    f"for {annotation_file} annotation_file_found={annotation_file_found}"
3116                                )
3117
3118                                # Full path
3119                                annotation_file_found = full_path(annotation_file_found)
3120
3121                                if annotation_file_found:
3122
3123                                    database = Database(database=annotation_file_found)
3124                                    quick_annotation_format = database.get_format()
3125                                    quick_annotation_is_compressed = (
3126                                        database.is_compressed()
3127                                    )
3128                                    quick_annotation_is_indexed = os.path.exists(
3129                                        f"{annotation_file_found}.tbi"
3130                                    )
3131                                    bcftools_preference = False
3132
3133                                    # Check Annotation Tool
3134                                    if not annotation_tool:
3135                                        if (
3136                                            bcftools_preference
3137                                            and quick_annotation_format
3138                                            in ["vcf", "bed"]
3139                                            and quick_annotation_is_compressed
3140                                            and quick_annotation_is_indexed
3141                                        ):
3142                                            annotation_tool = "bcftools"
3143                                        elif quick_annotation_format in [
3144                                            "vcf",
3145                                            "bed",
3146                                            "tsv",
3147                                            "tsv",
3148                                            "csv",
3149                                            "json",
3150                                            "tbl",
3151                                            "parquet",
3152                                            "duckdb",
3153                                        ]:
3154                                            annotation_tool = "parquet"
3155                                        else:
3156                                            log.error(
3157                                                f"Quick Annotation File {annotation_file_found} - Format {quick_annotation_format} not supported yet"
3158                                            )
3159                                            raise ValueError(
3160                                                f"Quick Annotation File {annotation_file_found} - Format {quick_annotation_format} not supported yet"
3161                                            )
3162
3163                                    log.debug(
3164                                        f"Quick Annotation File {annotation_file} - Annotation tool: {annotation_tool}"
3165                                    )
3166
3167                                    # Annotation Tool dispatch
3168                                    if annotation_tool:
3169                                        if annotation_tool not in param["annotation"]:
3170                                            param["annotation"][annotation_tool] = {}
3171                                        if (
3172                                            "annotations"
3173                                            not in param["annotation"][annotation_tool]
3174                                        ):
3175                                            param["annotation"][annotation_tool][
3176                                                "annotations"
3177                                            ] = {}
3178                                        param["annotation"][annotation_tool][
3179                                            "annotations"
3180                                        ][annotation_file_found] = annotations
3181
3182                                else:
3183                                    log.error(
3184                                        f"Quick Annotation File {annotation_file} does NOT exist"
3185                                    )
3186
3187                self.set_param(param)
3188
3189        if param.get("annotation", None):
3190            log.info("Annotations")
3191            if param.get("annotation", {}).get("parquet", None):
3192                log.info("Annotations 'parquet'...")
3193                self.annotation_parquet()
3194            if param.get("annotation", {}).get("bcftools", None):
3195                log.info("Annotations 'bcftools'...")
3196                self.annotation_bcftools()
3197            if param.get("annotation", {}).get("snpsift", None):
3198                log.info("Annotations 'snpsift'...")
3199                self.annotation_snpsift()
3200            if param.get("annotation", {}).get("annovar", None):
3201                log.info("Annotations 'annovar'...")
3202                self.annotation_annovar()
3203            if param.get("annotation", {}).get("snpeff", None):
3204                log.info("Annotations 'snpeff'...")
3205                self.annotation_snpeff()
3206            if param.get("annotation", {}).get("exomiser", None) is not None:
3207                log.info("Annotations 'exomiser'...")
3208                self.annotation_exomiser()
3209            if param.get("annotation", {}).get("splice", None) is not None:
3210                log.info("Annotations 'splice' ...")
3211                self.annotation_splice()
3212
3213        # Explode INFOS fields into table fields
3214        if self.get_explode_infos():
3215            self.explode_infos(
3216                prefix=self.get_explode_infos_prefix(),
3217                fields=self.get_explode_infos_fields(),
3218                force=True,
3219            )

It annotates the VCF file with the annotations specified in the config file.

def annotation_snpsift(self, threads: int = None) -> None: View Source

3221    def annotation_snpsift(self, threads: int = None) -> None:
3222        """
3223        This function annotate with bcftools
3224
3225        :param threads: Number of threads to use
3226        :return: the value of the variable "return_value".
3227        """
3228
3229        # DEBUG
3230        log.debug("Start annotation with bcftools databases")
3231
3232        # Threads
3233        if not threads:
3234            threads = self.get_threads()
3235        log.debug("Threads: " + str(threads))
3236
3237        # Config
3238        config = self.get_config()
3239        log.debug("Config: " + str(config))
3240
3241        # Config - snpSift
3242        snpsift_bin_command = get_bin_command(
3243            bin="SnpSift.jar",
3244            tool="snpsift",
3245            bin_type="jar",
3246            config=config,
3247            default_folder=f"{DEFAULT_TOOLS_FOLDER}/snpeff",
3248        )
3249        if not snpsift_bin_command:
3250            msg_err = f"Annotation failed: no snpsift bin '{snpsift_bin_command}'"
3251            log.error(msg_err)
3252            raise ValueError(msg_err)
3253
3254        # Config - bcftools
3255        bcftools_bin_command = get_bin_command(
3256            bin="bcftools",
3257            tool="bcftools",
3258            bin_type="bin",
3259            config=config,
3260            default_folder=f"{DEFAULT_TOOLS_FOLDER}/bcftools",
3261        )
3262        if not bcftools_bin_command:
3263            msg_err = f"Annotation failed: no bcftools bin '{bcftools_bin_command}'"
3264            log.error(msg_err)
3265            raise ValueError(msg_err)
3266
3267        # Config - BCFTools databases folders
3268        databases_folders = set(
3269            self.get_config()
3270            .get("folders", {})
3271            .get("databases", {})
3272            .get("annotations", ["."])
3273            + self.get_config()
3274            .get("folders", {})
3275            .get("databases", {})
3276            .get("bcftools", ["."])
3277        )
3278        log.debug("Databases annotations: " + str(databases_folders))
3279
3280        # Param
3281        annotations = (
3282            self.get_param()
3283            .get("annotation", {})
3284            .get("snpsift", {})
3285            .get("annotations", None)
3286        )
3287        log.debug("Annotations: " + str(annotations))
3288
3289        # Assembly
3290        assembly = self.get_param().get(
3291            "assembly", self.get_config().get("assembly", DEFAULT_ASSEMBLY)
3292        )
3293
3294        # Data
3295        table_variants = self.get_table_variants()
3296
3297        # Check if not empty
3298        log.debug("Check if not empty")
3299        sql_query_chromosomes = (
3300            f"""SELECT count(*) as count FROM {table_variants} as table_variants"""
3301        )
3302        sql_query_chromosomes_df = self.get_query_to_df(sql_query_chromosomes)
3303        if not sql_query_chromosomes_df["count"][0]:
3304            log.info(f"VCF empty")
3305            return
3306
3307        # VCF header
3308        vcf_reader = self.get_header()
3309        log.debug("Initial header: " + str(vcf_reader.infos))
3310
3311        # Existing annotations
3312        for vcf_annotation in self.get_header().infos:
3313
3314            vcf_annotation_line = self.get_header().infos.get(vcf_annotation)
3315            log.debug(
3316                f"Existing annotations in VCF: {vcf_annotation} [{vcf_annotation_line}]"
3317            )
3318
3319        if annotations:
3320
3321            with TemporaryDirectory(dir=self.get_tmp_dir()) as tmp_dir:
3322
3323                # Export VCF file
3324                tmp_vcf_name = os.path.join(tmp_dir, "input.vcf.gz")
3325
3326                # Init
3327                commands = {}
3328
3329                for annotation in annotations:
3330                    annotation_fields = annotations[annotation]
3331
3332                    # Annotation Name
3333                    annotation_name = os.path.basename(annotation)
3334
3335                    if not annotation_fields:
3336                        annotation_fields = {"INFO": None}
3337
3338                    log.debug(f"Annotation '{annotation_name}'")
3339                    log.debug(
3340                        f"Annotation '{annotation_name}' - fields: {annotation_fields}"
3341                    )
3342
3343                    # Create Database
3344                    database = Database(
3345                        database=annotation,
3346                        databases_folders=databases_folders,
3347                        assembly=assembly,
3348                    )
3349
3350                    # Find files
3351                    db_file = database.get_database()
3352                    db_file = full_path(db_file)
3353                    db_hdr_file = database.get_header_file()
3354                    db_hdr_file = full_path(db_hdr_file)
3355                    db_file_type = database.get_format()
3356                    db_tbi_file = f"{db_file}.tbi"
3357                    db_file_compressed = database.is_compressed()
3358
3359                    # Check if compressed
3360                    if not db_file_compressed:
3361                        log.error(
3362                            f"Annotation '{annotation}' - {db_file} NOT compressed file"
3363                        )
3364                        raise ValueError(
3365                            f"Annotation '{annotation}' - {db_file} NOT compressed file"
3366                        )
3367
3368                    # Check if indexed
3369                    if not os.path.exists(db_tbi_file):
3370                        log.error(
3371                            f"Annotation '{annotation}' - {db_file} NOT indexed file"
3372                        )
3373                        raise ValueError(
3374                            f"Annotation '{annotation}' - {db_file} NOT indexed file"
3375                        )
3376
3377                    # Check index - try to create if not exists
3378                    if not os.path.exists(db_file) or not os.path.exists(db_hdr_file):
3379                        log.error("Annotation failed: database not valid")
3380                        log.error(f"Annotation annotation file: {db_file}")
3381                        log.error(f"Annotation annotation header: {db_hdr_file}")
3382                        log.error(f"Annotation annotation index: {db_tbi_file}")
3383                        raise ValueError(
3384                            f"Annotation failed: database not valid - annotation file {db_file} / annotation header {db_hdr_file} / annotation index {db_tbi_file} / annotation compression {db_file_compressed}"
3385                        )
3386                    else:
3387
3388                        log.debug(
3389                            f"Annotation '{annotation}' - file: "
3390                            + str(db_file)
3391                            + " and "
3392                            + str(db_hdr_file)
3393                        )
3394
3395                        # Load header as VCF object
3396                        db_hdr_vcf = Variants(input=db_hdr_file)
3397                        db_hdr_vcf_header_infos = db_hdr_vcf.get_header().infos
3398                        log.debug(
3399                            "Annotation database header: "
3400                            + str(db_hdr_vcf_header_infos)
3401                        )
3402
3403                        # For all fields in database
3404                        annotation_fields_full = False
3405                        if "ALL" in annotation_fields or "INFO" in annotation_fields:
3406                            annotation_fields = {
3407                                key: key for key in db_hdr_vcf_header_infos
3408                            }
3409                            log.debug(
3410                                "Annotation database header - All annotations added: "
3411                                + str(annotation_fields)
3412                            )
3413                            annotation_fields_full = True
3414
3415                        # # Create file for field rename
3416                        # log.debug("Create file for field rename")
3417                        # tmp_rename = NamedTemporaryFile(
3418                        #     prefix=self.get_prefix(),
3419                        #     dir=self.get_tmp_dir(),
3420                        #     suffix=".rename",
3421                        #     delete=False,
3422                        # )
3423                        # tmp_rename_name = tmp_rename.name
3424                        # tmp_files.append(tmp_rename_name)
3425
3426                        # Number of fields
3427                        nb_annotation_field = 0
3428                        annotation_list = []
3429                        annotation_infos_rename_list = []
3430
3431                        for annotation_field in annotation_fields:
3432
3433                            # field new name, if parametered SKIPPED !!!!!! not managed actually TODO
3434                            annotation_fields_new_name = annotation_fields.get(
3435                                annotation_field, annotation_field
3436                            )
3437                            if not annotation_fields_new_name:
3438                                annotation_fields_new_name = annotation_field
3439
3440                            # Check if field is in DB and if field is not elready in input data
3441                            if (
3442                                annotation_field in db_hdr_vcf.get_header().infos
3443                                and annotation_fields_new_name
3444                                not in self.get_header().infos
3445                            ):
3446
3447                                log.info(
3448                                    f"Annotation '{annotation_name}' - '{annotation_field}' -> '{annotation_fields_new_name}'"
3449                                )
3450
3451                                # BCFTools annotate param to rename fields
3452                                if annotation_field != annotation_fields_new_name:
3453                                    annotation_infos_rename_list.append(
3454                                        f"{annotation_fields_new_name}:=INFO/{annotation_field}"
3455                                    )
3456
3457                                # Add INFO field to header
3458                                db_hdr_vcf_header_infos_number = (
3459                                    db_hdr_vcf_header_infos[annotation_field].num or "."
3460                                )
3461                                db_hdr_vcf_header_infos_type = (
3462                                    db_hdr_vcf_header_infos[annotation_field].type
3463                                    or "String"
3464                                )
3465                                db_hdr_vcf_header_infos_description = (
3466                                    db_hdr_vcf_header_infos[annotation_field].desc
3467                                    or f"{annotation_field} description"
3468                                )
3469                                db_hdr_vcf_header_infos_source = (
3470                                    db_hdr_vcf_header_infos[annotation_field].source
3471                                    or "unknown"
3472                                )
3473                                db_hdr_vcf_header_infos_version = (
3474                                    db_hdr_vcf_header_infos[annotation_field].version
3475                                    or "unknown"
3476                                )
3477
3478                                vcf_reader.infos[annotation_fields_new_name] = (
3479                                    vcf.parser._Info(
3480                                        annotation_fields_new_name,
3481                                        db_hdr_vcf_header_infos_number,
3482                                        db_hdr_vcf_header_infos_type,
3483                                        db_hdr_vcf_header_infos_description,
3484                                        db_hdr_vcf_header_infos_source,
3485                                        db_hdr_vcf_header_infos_version,
3486                                        self.code_type_map[
3487                                            db_hdr_vcf_header_infos_type
3488                                        ],
3489                                    )
3490                                )
3491
3492                                annotation_list.append(annotation_field)
3493
3494                                nb_annotation_field += 1
3495
3496                            else:
3497
3498                                if (
3499                                    annotation_field
3500                                    not in db_hdr_vcf.get_header().infos
3501                                ):
3502                                    log.warning(
3503                                        f"Annotation '{annotation_name}' - '{annotation_field}' - not available in vcf/bed file"
3504                                    )
3505                                if (
3506                                    annotation_fields_new_name
3507                                    in self.get_header().infos
3508                                ):
3509                                    log.warning(
3510                                        f"Annotation '{annotation_name}' - '{annotation_fields_new_name}' - already exists (skipped)"
3511                                    )
3512
3513                        log.info(
3514                            f"Annotation '{annotation_name}' - {nb_annotation_field} annotations available in vcf/bed file"
3515                        )
3516
3517                        annotation_infos = ",".join(annotation_list)
3518
3519                        if annotation_infos != "":
3520
3521                            # Annotated VCF (and error file)
3522                            tmp_annotation_vcf_name = os.path.join(
3523                                tmp_dir, os.path.basename(annotation) + ".vcf.gz"
3524                            )
3525                            tmp_annotation_vcf_name_err = (
3526                                tmp_annotation_vcf_name + ".err"
3527                            )
3528
3529                            # Add fields to annotate
3530                            if not annotation_fields_full:
3531                                annotation_infos_option = f"-info {annotation_infos}"
3532                            else:
3533                                annotation_infos_option = ""
3534
3535                            # Info fields rename
3536                            if annotation_infos_rename_list:
3537                                annotation_infos_rename = " -c " + ",".join(
3538                                    annotation_infos_rename_list
3539                                )
3540                            else:
3541                                annotation_infos_rename = ""
3542
3543                            # Annotate command
3544                            command_annotate = f"{snpsift_bin_command} annotate {annotation_infos_option} {db_file} {tmp_vcf_name} | {bcftools_bin_command} annotate --threads={threads} {annotation_infos_rename} -Oz1 -o {tmp_annotation_vcf_name} 2>>{tmp_annotation_vcf_name_err} "
3545
3546                            # Add command
3547                            commands[command_annotate] = tmp_annotation_vcf_name
3548
3549                if commands:
3550
3551                    # Export VCF file
3552                    self.export_variant_vcf(
3553                        vcf_file=tmp_vcf_name,
3554                        remove_info=True,
3555                        add_samples=False,
3556                        index=True,
3557                    )
3558                    shutil.copyfile(tmp_vcf_name, "/tmp/input.vcf")
3559
3560                    # Num command
3561                    nb_command = 0
3562
3563                    # Annotate
3564                    for command_annotate in commands:
3565                        nb_command += 1
3566                        log.info(
3567                            f"Annotation - Annotate [{nb_command}/{len(commands)}]..."
3568                        )
3569                        log.debug(f"command_annotate={command_annotate}")
3570                        run_parallel_commands([command_annotate], threads)
3571
3572                        # Debug
3573                        shutil.copyfile(commands[command_annotate], "/tmp/snpsift.vcf")
3574
3575                        # Update variants
3576                        log.info(
3577                            f"Annotation - Updating [{nb_command}/{len(commands)}]..."
3578                        )
3579                        self.update_from_vcf(commands[command_annotate])

This function annotate with bcftools

Parameters

threads: Number of threads to use

Returns

the value of the variable "return_value".

def annotation_bcftools(self, threads: int = None) -> None: View Source

3581    def annotation_bcftools(self, threads: int = None) -> None:
3582        """
3583        This function annotate with bcftools
3584
3585        :param threads: Number of threads to use
3586        :return: the value of the variable "return_value".
3587        """
3588
3589        # DEBUG
3590        log.debug("Start annotation with bcftools databases")
3591
3592        # Threads
3593        if not threads:
3594            threads = self.get_threads()
3595        log.debug("Threads: " + str(threads))
3596
3597        # Config
3598        config = self.get_config()
3599        log.debug("Config: " + str(config))
3600
3601        # DEBUG
3602        delete_tmp = True
3603        if self.get_config().get("verbosity", "warning") in ["debug"]:
3604            delete_tmp = False
3605            log.debug("Delete tmp files/folders: " + str(delete_tmp))
3606
3607        # Config - BCFTools bin command
3608        bcftools_bin_command = get_bin_command(
3609            bin="bcftools",
3610            tool="bcftools",
3611            bin_type="bin",
3612            config=config,
3613            default_folder=f"{DEFAULT_TOOLS_FOLDER}/bcftools",
3614        )
3615        if not bcftools_bin_command:
3616            msg_err = f"Annotation failed: no bcftools bin '{bcftools_bin_command}'"
3617            log.error(msg_err)
3618            raise ValueError(msg_err)
3619
3620        # Config - BCFTools databases folders
3621        databases_folders = set(
3622            self.get_config()
3623            .get("folders", {})
3624            .get("databases", {})
3625            .get("annotations", ["."])
3626            + self.get_config()
3627            .get("folders", {})
3628            .get("databases", {})
3629            .get("bcftools", ["."])
3630        )
3631        log.debug("Databases annotations: " + str(databases_folders))
3632
3633        # Param
3634        annotations = (
3635            self.get_param()
3636            .get("annotation", {})
3637            .get("bcftools", {})
3638            .get("annotations", None)
3639        )
3640        log.debug("Annotations: " + str(annotations))
3641
3642        # Assembly
3643        assembly = self.get_param().get(
3644            "assembly", self.get_config().get("assembly", DEFAULT_ASSEMBLY)
3645        )
3646
3647        # Data
3648        table_variants = self.get_table_variants()
3649
3650        # Check if not empty
3651        log.debug("Check if not empty")
3652        sql_query_chromosomes = (
3653            f"""SELECT count(*) as count FROM {table_variants} as table_variants"""
3654        )
3655        sql_query_chromosomes_df = self.get_query_to_df(sql_query_chromosomes)
3656        if not sql_query_chromosomes_df["count"][0]:
3657            log.info(f"VCF empty")
3658            return
3659
3660        # Export in VCF
3661        log.debug("Create initial file to annotate")
3662        tmp_vcf = NamedTemporaryFile(
3663            prefix=self.get_prefix(),
3664            dir=self.get_tmp_dir(),
3665            suffix=".vcf.gz",
3666            delete=False,
3667        )
3668        tmp_vcf_name = tmp_vcf.name
3669
3670        # VCF header
3671        vcf_reader = self.get_header()
3672        log.debug("Initial header: " + str(vcf_reader.infos))
3673
3674        # Existing annotations
3675        for vcf_annotation in self.get_header().infos:
3676
3677            vcf_annotation_line = self.get_header().infos.get(vcf_annotation)
3678            log.debug(
3679                f"Existing annotations in VCF: {vcf_annotation} [{vcf_annotation_line}]"
3680            )
3681
3682        if annotations:
3683
3684            tmp_ann_vcf_list = []
3685            commands = []
3686            tmp_files = []
3687            err_files = []
3688
3689            for annotation in annotations:
3690                annotation_fields = annotations[annotation]
3691
3692                # Annotation Name
3693                annotation_name = os.path.basename(annotation)
3694
3695                if not annotation_fields:
3696                    annotation_fields = {"INFO": None}
3697
3698                log.debug(f"Annotation '{annotation_name}'")
3699                log.debug(
3700                    f"Annotation '{annotation_name}' - fields: {annotation_fields}"
3701                )
3702
3703                # Create Database
3704                database = Database(
3705                    database=annotation,
3706                    databases_folders=databases_folders,
3707                    assembly=assembly,
3708                )
3709
3710                # Find files
3711                db_file = database.get_database()
3712                db_file = full_path(db_file)
3713                db_hdr_file = database.get_header_file()
3714                db_hdr_file = full_path(db_hdr_file)
3715                db_file_type = database.get_format()
3716                db_tbi_file = f"{db_file}.tbi"
3717                db_file_compressed = database.is_compressed()
3718
3719                # Check if compressed
3720                if not db_file_compressed:
3721                    log.error(
3722                        f"Annotation '{annotation}' - {db_file} NOT compressed file"
3723                    )
3724                    raise ValueError(
3725                        f"Annotation '{annotation}' - {db_file} NOT compressed file"
3726                    )
3727
3728                # Check if indexed
3729                if not os.path.exists(db_tbi_file):
3730                    log.error(f"Annotation '{annotation}' - {db_file} NOT indexed file")
3731                    raise ValueError(
3732                        f"Annotation '{annotation}' - {db_file} NOT indexed file"
3733                    )
3734
3735                # Check index - try to create if not exists
3736                if not os.path.exists(db_file) or not os.path.exists(db_hdr_file):
3737                    log.error("Annotation failed: database not valid")
3738                    log.error(f"Annotation annotation file: {db_file}")
3739                    log.error(f"Annotation annotation header: {db_hdr_file}")
3740                    log.error(f"Annotation annotation index: {db_tbi_file}")
3741                    raise ValueError(
3742                        f"Annotation failed: database not valid - annotation file {db_file} / annotation header {db_hdr_file} / annotation index {db_tbi_file} / annotation compression {db_file_compressed}"
3743                    )
3744                else:
3745
3746                    log.debug(
3747                        f"Annotation '{annotation}' - file: "
3748                        + str(db_file)
3749                        + " and "
3750                        + str(db_hdr_file)
3751                    )
3752
3753                    # Load header as VCF object
3754                    db_hdr_vcf = Variants(input=db_hdr_file)
3755                    db_hdr_vcf_header_infos = db_hdr_vcf.get_header().infos
3756                    log.debug(
3757                        "Annotation database header: " + str(db_hdr_vcf_header_infos)
3758                    )
3759
3760                    # For all fields in database
3761                    if "ALL" in annotation_fields or "INFO" in annotation_fields:
3762                        annotation_fields = {
3763                            key: key for key in db_hdr_vcf_header_infos
3764                        }
3765                        log.debug(
3766                            "Annotation database header - All annotations added: "
3767                            + str(annotation_fields)
3768                        )
3769
3770                    # Number of fields
3771                    nb_annotation_field = 0
3772                    annotation_list = []
3773
3774                    for annotation_field in annotation_fields:
3775
3776                        # field new name, if parametered SKIPPED !!!!!! not managed actually TODO
3777                        annotation_fields_new_name = annotation_fields.get(
3778                            annotation_field, annotation_field
3779                        )
3780                        if not annotation_fields_new_name:
3781                            annotation_fields_new_name = annotation_field
3782
3783                        # Check if field is in DB and if field is not elready in input data
3784                        if (
3785                            annotation_field in db_hdr_vcf.get_header().infos
3786                            and annotation_fields_new_name
3787                            not in self.get_header().infos
3788                        ):
3789
3790                            log.info(
3791                                f"Annotation '{annotation_name}' - '{annotation_field}' -> '{annotation_fields_new_name}'"
3792                            )
3793
3794                            # Add INFO field to header
3795                            db_hdr_vcf_header_infos_number = (
3796                                db_hdr_vcf_header_infos[annotation_field].num or "."
3797                            )
3798                            db_hdr_vcf_header_infos_type = (
3799                                db_hdr_vcf_header_infos[annotation_field].type
3800                                or "String"
3801                            )
3802                            db_hdr_vcf_header_infos_description = (
3803                                db_hdr_vcf_header_infos[annotation_field].desc
3804                                or f"{annotation_field} description"
3805                            )
3806                            db_hdr_vcf_header_infos_source = (
3807                                db_hdr_vcf_header_infos[annotation_field].source
3808                                or "unknown"
3809                            )
3810                            db_hdr_vcf_header_infos_version = (
3811                                db_hdr_vcf_header_infos[annotation_field].version
3812                                or "unknown"
3813                            )
3814
3815                            vcf_reader.infos[annotation_fields_new_name] = (
3816                                vcf.parser._Info(
3817                                    annotation_fields_new_name,
3818                                    db_hdr_vcf_header_infos_number,
3819                                    db_hdr_vcf_header_infos_type,
3820                                    db_hdr_vcf_header_infos_description,
3821                                    db_hdr_vcf_header_infos_source,
3822                                    db_hdr_vcf_header_infos_version,
3823                                    self.code_type_map[db_hdr_vcf_header_infos_type],
3824                                )
3825                            )
3826
3827                            # annotation_list.append(annotation_field)
3828                            if annotation_field != annotation_fields_new_name:
3829                                annotation_list.append(
3830                                    f"{annotation_fields_new_name}:=INFO/{annotation_field}"
3831                                )
3832                            else:
3833                                annotation_list.append(annotation_field)
3834
3835                            nb_annotation_field += 1
3836
3837                        else:
3838
3839                            if annotation_field not in db_hdr_vcf.get_header().infos:
3840                                log.warning(
3841                                    f"Annotation '{annotation}' - '{annotation_field}' - not available in vcf/bed file"
3842                                )
3843                            if annotation_fields_new_name in self.get_header().infos:
3844                                log.warning(
3845                                    f"Annotation '{annotation}' - '{annotation_fields_new_name}' - already exists (skipped)"
3846                                )
3847
3848                    log.info(
3849                        f"Annotation '{annotation_name}' - {nb_annotation_field} annotations available in vcf/bed file"
3850                    )
3851
3852                    annotation_infos = ",".join(annotation_list)
3853
3854                    if annotation_infos != "":
3855
3856                        # Protect header for bcftools (remove "#CHROM" and variants line)
3857                        log.debug("Protect Header file - remove #CHROM line if exists")
3858                        tmp_header_vcf = NamedTemporaryFile(
3859                            prefix=self.get_prefix(),
3860                            dir=self.get_tmp_dir(),
3861                            suffix=".hdr",
3862                            delete=False,
3863                        )
3864                        tmp_header_vcf_name = tmp_header_vcf.name
3865                        tmp_files.append(tmp_header_vcf_name)
3866                        # Command
3867                        if db_hdr_file.endswith(".gz"):
3868                            command_extract_header = f"zcat {db_hdr_file} | grep '^##' > {tmp_header_vcf_name}"
3869                        else:
3870                            command_extract_header = f"cat {db_hdr_file} | grep '^##' > {tmp_header_vcf_name}"
3871                        # Run
3872                        run_parallel_commands([command_extract_header], 1)
3873
3874                        # Find chomosomes
3875                        log.debug("Find chromosomes ")
3876                        sql_query_chromosomes = f"""SELECT table_variants.\"#CHROM\" as CHROM FROM {table_variants} as table_variants GROUP BY table_variants.\"#CHROM\""""
3877                        sql_query_chromosomes_df = self.get_query_to_df(
3878                            sql_query_chromosomes
3879                        )
3880                        chomosomes_list = list(sql_query_chromosomes_df["CHROM"])
3881
3882                        log.debug("Chromosomes found: " + str(list(chomosomes_list)))
3883
3884                        # BED columns in the annotation file
3885                        if db_file_type in ["bed"]:
3886                            annotation_infos = "CHROM,POS,POS," + annotation_infos
3887
3888                        for chrom in chomosomes_list:
3889
3890                            # Create BED on initial VCF
3891                            log.debug("Create BED on initial VCF: " + str(tmp_vcf_name))
3892                            tmp_bed = NamedTemporaryFile(
3893                                prefix=self.get_prefix(),
3894                                dir=self.get_tmp_dir(),
3895                                suffix=".bed",
3896                                delete=False,
3897                            )
3898                            tmp_bed_name = tmp_bed.name
3899                            tmp_files.append(tmp_bed_name)
3900
3901                            # Detecte regions
3902                            log.debug(
3903                                f"Annotation '{annotation}' - Chromosome '{chrom}' - Start detecting regions..."
3904                            )
3905                            window = 1000000
3906                            sql_query_intervals_for_bed = f"""
3907                                SELECT  \"#CHROM\",
3908                                        CASE WHEN \"POS\"-{window}-1 < 0 THEN 0 ELSE \"POS\"-{window}-1 END,
3909                                        \"POS\"+{window}
3910                                FROM {table_variants} as table_variants
3911                                WHERE table_variants.\"#CHROM\" = '{chrom}'
3912                            """
3913                            regions = self.conn.execute(
3914                                sql_query_intervals_for_bed
3915                            ).fetchall()
3916                            merged_regions = merge_regions(regions)
3917                            log.debug(
3918                                f"Annotation '{annotation}' - Chromosome '{chrom}' - Stop detecting regions..."
3919                            )
3920
3921                            header = ["#CHROM", "START", "END"]
3922                            with open(tmp_bed_name, "w") as f:
3923                                # Write the header with tab delimiter
3924                                f.write("\t".join(header) + "\n")
3925                                for d in merged_regions:
3926                                    # Write each data row with tab delimiter
3927                                    f.write("\t".join(map(str, d)) + "\n")
3928
3929                            # Tmp files
3930                            tmp_annotation_vcf = NamedTemporaryFile(
3931                                prefix=self.get_prefix(),
3932                                dir=self.get_tmp_dir(),
3933                                suffix=".vcf.gz",
3934                                delete=False,
3935                            )
3936                            tmp_annotation_vcf_name = tmp_annotation_vcf.name
3937                            tmp_files.append(tmp_annotation_vcf_name)
3938                            tmp_ann_vcf_list.append(f"{tmp_annotation_vcf_name}")
3939                            tmp_annotation_vcf_name_err = (
3940                                tmp_annotation_vcf_name + ".err"
3941                            )
3942                            err_files.append(tmp_annotation_vcf_name_err)
3943
3944                            # Annotate Command
3945                            log.debug(
3946                                f"Annotation '{annotation}' - add bcftools command"
3947                            )
3948
3949                            # Command
3950                            command_annotate = f"{bcftools_bin_command} annotate --pair-logic exact --regions-file={tmp_bed_name} -a {db_file} -h {tmp_header_vcf_name} -c {annotation_infos} {tmp_vcf_name} -o {tmp_annotation_vcf_name} -Oz1 2>>{tmp_annotation_vcf_name_err} && tabix {tmp_annotation_vcf_name} 2>>{tmp_annotation_vcf_name_err} "
3951
3952                            # Add command
3953                            commands.append(command_annotate)
3954
3955            # if some commands
3956            if commands:
3957
3958                # Export VCF file
3959                self.export_variant_vcf(
3960                    vcf_file=tmp_vcf_name,
3961                    remove_info=True,
3962                    add_samples=False,
3963                    index=True,
3964                )
3965
3966                # Threads
3967                # calculate threads for annotated commands
3968                if commands:
3969                    threads_bcftools_annotate = round(threads / len(commands))
3970                else:
3971                    threads_bcftools_annotate = 1
3972
3973                if not threads_bcftools_annotate:
3974                    threads_bcftools_annotate = 1
3975
3976                # Add threads option to bcftools commands
3977                if threads_bcftools_annotate > 1:
3978                    commands_threaded = []
3979                    for command in commands:
3980                        commands_threaded.append(
3981                            command.replace(
3982                                f"{bcftools_bin_command} annotate ",
3983                                f"{bcftools_bin_command} annotate --threads={threads_bcftools_annotate} ",
3984                            )
3985                        )
3986                    commands = commands_threaded
3987
3988                # Command annotation multithreading
3989                log.debug(f"Annotation - Annotation commands: " + str(commands))
3990                log.info(
3991                    f"Annotation - Annotation multithreaded in "
3992                    + str(len(commands))
3993                    + " commands"
3994                )
3995
3996                run_parallel_commands(commands, threads)
3997
3998                # Merge
3999                tmp_ann_vcf_list_cmd = " ".join(tmp_ann_vcf_list)
4000
4001                if tmp_ann_vcf_list_cmd:
4002
4003                    # Tmp file
4004                    tmp_annotate_vcf = NamedTemporaryFile(
4005                        prefix=self.get_prefix(),
4006                        dir=self.get_tmp_dir(),
4007                        suffix=".vcf.gz",
4008                        delete=True,
4009                    )
4010                    tmp_annotate_vcf_name = tmp_annotate_vcf.name
4011                    tmp_annotate_vcf_name_err = tmp_annotate_vcf_name + ".err"
4012                    err_files.append(tmp_annotate_vcf_name_err)
4013
4014                    # Tmp file remove command
4015                    tmp_files_remove_command = ""
4016                    if tmp_files:
4017                        tmp_files_remove_command = " && rm -f " + " ".join(tmp_files)
4018
4019                    # Command merge
4020                    merge_command = f"{bcftools_bin_command} merge --force-samples --threads={threads} {tmp_vcf_name} {tmp_ann_vcf_list_cmd} -o {tmp_annotate_vcf_name} -Oz 2>>{tmp_annotate_vcf_name_err} {tmp_files_remove_command}"
4021                    log.info(
4022                        f"Annotation - Annotation merging "
4023                        + str(len(commands))
4024                        + " annotated files"
4025                    )
4026                    log.debug(f"Annotation - merge command: {merge_command}")
4027                    run_parallel_commands([merge_command], 1)
4028
4029                    # Error messages
4030                    log.info(f"Error/Warning messages:")
4031                    error_message_command_all = []
4032                    error_message_command_warning = []
4033                    error_message_command_err = []
4034                    for err_file in err_files:
4035                        with open(err_file, "r") as f:
4036                            for line in f:
4037                                message = line.strip()
4038                                error_message_command_all.append(message)
4039                                if line.startswith("[W::"):
4040                                    error_message_command_warning.append(message)
4041                                if line.startswith("[E::"):
4042                                    error_message_command_err.append(
4043                                        f"{err_file}: " + message
4044                                    )
4045                    # log info
4046                    for message in list(
4047                        set(error_message_command_err + error_message_command_warning)
4048                    ):
4049                        log.info(f"   {message}")
4050                    # debug info
4051                    for message in list(set(error_message_command_all)):
4052                        log.debug(f"   {message}")
4053                    # failed
4054                    if len(error_message_command_err):
4055                        log.error("Annotation failed: Error in commands")
4056                        raise ValueError("Annotation failed: Error in commands")
4057
4058                    # Update variants
4059                    log.info(f"Annotation - Updating...")
4060                    self.update_from_vcf(tmp_annotate_vcf_name)

This function annotate with bcftools

Parameters

threads: Number of threads to use

Returns

the value of the variable "return_value".

def annotation_exomiser(self, threads: int = None) -> None: View Source

4062    def annotation_exomiser(self, threads: int = None) -> None:
4063        """
4064        This function annotate with Exomiser
4065
4066        This function uses args as parameters, in section "annotation" -> "exomiser", with sections:
4067        - "analysis" (dict/file):
4068            Full analysis dictionnary parameters (see Exomiser docs).
4069            Either a dict, or a file in JSON or YAML format.
4070            These parameters may change depending on other parameters (e.g. phenotipicFeatures/HPO)
4071            Default : None
4072        - "preset" (string):
4073            Analysis preset (available in config folder).
4074            Used if no full "analysis" is provided.
4075            Default: "exome"
4076        - "phenopacket" (dict/file):
4077            Samples and phenotipic features parameters (see Exomiser docs).
4078            Either a dict, or a file in JSON or YAML format.
4079            Default: None
4080        - "subject" (dict):
4081            Sample parameters (see Exomiser docs).
4082            Example:
4083                "subject":
4084                    {
4085                        "id": "ISDBM322017",
4086                        "sex": "FEMALE"
4087                    }
4088            Default: None
4089        - "sample" (string):
4090            Sample name to construct "subject" section:
4091                "subject":
4092                    {
4093                        "id": "<sample>",
4094                        "sex": "UNKNOWN_SEX"
4095                    }
4096            Default: None
4097        - "phenotypicFeatures" (dict)
4098            Phenotypic features to construct "subject" section.
4099            Example:
4100                "phenotypicFeatures":
4101                    [
4102                        { "type": { "id": "HP:0001159", "label": "Syndactyly" } },
4103                        { "type": { "id": "HP:0000486", "label": "Strabismus" } }
4104                    ]
4105        - "hpo" (list)
4106            List of HPO ids as phenotypic features.
4107            Example:
4108                "hpo": ['0001156', '0001363', '0011304', '0010055']
4109            Default: []
4110        - "outputOptions" (dict):
4111            Output options (see Exomiser docs).
4112            Default:
4113                "output_options" =
4114                    {
4115                        "outputContributingVariantsOnly": False,
4116                        "numGenes": 0,
4117                        "outputFormats": ["TSV_VARIANT", "VCF"]
4118                    }
4119        - "transcript_source" (string):
4120            Transcript source (either "refseq", "ucsc", "ensembl")
4121            Default: "refseq"
4122        - "exomiser_to_info" (boolean):
4123            Add exomiser TSV file columns as INFO fields in VCF.
4124            Default: False
4125        - "release" (string):
4126            Exomise database release.
4127            If not exists, database release will be downloaded (take a while).
4128            Default: None (provided by application.properties configuration file)
4129        - "exomiser_application_properties" (file):
4130            Exomiser configuration file (see Exomiser docs).
4131            Useful to automatically download databases (especially for specific genome databases).
4132
4133        Notes:
4134        - If no sample in parameters, first sample in VCF will be chosen
4135        - If no HPO found, "hiPhivePrioritiser" analysis step will be switch off
4136
4137        :param threads: The number of threads to use
4138        :return: None.
4139        """
4140
4141        # DEBUG
4142        log.debug("Start annotation with Exomiser databases")
4143
4144        # Threads
4145        if not threads:
4146            threads = self.get_threads()
4147        log.debug("Threads: " + str(threads))
4148
4149        # Config
4150        config = self.get_config()
4151        log.debug("Config: " + str(config))
4152
4153        # Config - Folders - Databases
4154        databases_folders = (
4155            config.get("folders", {})
4156            .get("databases", {})
4157            .get("exomiser", f"{DEFAULT_DATABASE_FOLDER}/exomiser/current")
4158        )
4159        databases_folders = full_path(databases_folders)
4160        if not os.path.exists(databases_folders):
4161            log.error(f"Databases annotations: {databases_folders} NOT found")
4162        log.debug("Databases annotations: " + str(databases_folders))
4163
4164        # Config - Exomiser
4165        exomiser_bin_command = get_bin_command(
4166            bin="exomiser-cli*.jar",
4167            tool="exomiser",
4168            bin_type="jar",
4169            config=config,
4170            default_folder=f"{DEFAULT_TOOLS_FOLDER}/exomiser",
4171        )
4172        log.debug("Exomiser bin command: " + str(exomiser_bin_command))
4173        if not exomiser_bin_command:
4174            msg_err = f"Annotation failed: no exomiser bin '{exomiser_bin_command}'"
4175            log.error(msg_err)
4176            raise ValueError(msg_err)
4177
4178        # Param
4179        param = self.get_param()
4180        log.debug("Param: " + str(param))
4181
4182        # Param - Exomiser
4183        param_exomiser = param.get("annotation", {}).get("exomiser", {})
4184        log.debug(f"Param Exomiser: {param_exomiser}")
4185
4186        # Param - Assembly
4187        assembly = param.get("assembly", config.get("assembly", DEFAULT_ASSEMBLY))
4188        log.debug("Assembly: " + str(assembly))
4189
4190        # Data
4191        table_variants = self.get_table_variants()
4192
4193        # Check if not empty
4194        log.debug("Check if not empty")
4195        sql_query_chromosomes = (
4196            f"""SELECT count(*) as count FROM {table_variants} as table_variants"""
4197        )
4198        if not self.get_query_to_df(f"{sql_query_chromosomes}")["count"][0]:
4199            log.info(f"VCF empty")
4200            return False
4201
4202        # VCF header
4203        vcf_reader = self.get_header()
4204        log.debug("Initial header: " + str(vcf_reader.infos))
4205
4206        # Samples
4207        samples = self.get_header_sample_list()
4208        if not samples:
4209            log.error("No Samples in VCF")
4210            return False
4211        log.debug(f"Samples: {samples}")
4212
4213        # Memory limit
4214        memory_limit = self.get_memory("8G")
4215        log.debug(f"memory_limit: {memory_limit}")
4216
4217        # Exomiser java options
4218        exomiser_java_options = (
4219            f" -Xmx{memory_limit} -XX:+UseParallelGC -XX:ParallelGCThreads={threads} "
4220        )
4221        log.debug(f"Exomiser java options: {exomiser_java_options}")
4222
4223        # Download Exomiser (if not exists)
4224        exomiser_release = param_exomiser.get("release", None)
4225        exomiser_application_properties = param_exomiser.get(
4226            "exomiser_application_properties", None
4227        )
4228        databases_download_exomiser(
4229            assemblies=[assembly],
4230            exomiser_folder=databases_folders,
4231            exomiser_release=exomiser_release,
4232            exomiser_phenotype_release=exomiser_release,
4233            exomiser_application_properties=exomiser_application_properties,
4234        )
4235
4236        # Force annotation
4237        force_update_annotation = True
4238
4239        if "Exomiser" not in self.get_header().infos or force_update_annotation:
4240            log.debug("Start annotation Exomiser")
4241
4242            with TemporaryDirectory(dir=self.get_tmp_dir()) as tmp_dir:
4243
4244                # tmp_dir = "/tmp/exomiser"
4245
4246                ### ANALYSIS ###
4247                ################
4248
4249                # Create analysis.json through analysis dict
4250                # either analysis in param or by default
4251                # depending on preset exome/genome)
4252
4253                # Init analysis dict
4254                param_exomiser_analysis_dict = {}
4255
4256                # analysis from param
4257                param_exomiser_analysis = param_exomiser.get("analysis", {})
4258                param_exomiser_analysis = full_path(param_exomiser_analysis)
4259
4260                # If analysis in param -> load anlaysis json
4261                if param_exomiser_analysis:
4262
4263                    # If param analysis is a file and exists
4264                    if isinstance(param_exomiser_analysis, str) and os.path.exists(
4265                        param_exomiser_analysis
4266                    ):
4267                        # Load analysis file into analysis dict (either yaml or json)
4268                        with open(param_exomiser_analysis) as json_file:
4269                            param_exomiser_analysis_dict = yaml.safe_load(json_file)
4270
4271                    # If param analysis is a dict
4272                    elif isinstance(param_exomiser_analysis, dict):
4273                        # Load analysis dict into analysis dict (either yaml or json)
4274                        param_exomiser_analysis_dict = param_exomiser_analysis
4275
4276                    # Error analysis type
4277                    else:
4278                        log.error(f"Analysis type unknown. Check param file.")
4279                        raise ValueError(f"Analysis type unknown. Check param file.")
4280
4281                # Case no input analysis config file/dict
4282                # Use preset (exome/genome) to open default config file
4283                if not param_exomiser_analysis_dict:
4284
4285                    # default preset
4286                    default_preset = "exome"
4287
4288                    # Get param preset or default preset
4289                    param_exomiser_preset = param_exomiser.get("preset", default_preset)
4290
4291                    # Try to find if preset is a file
4292                    if os.path.exists(param_exomiser_preset):
4293                        # Preset file is provided in full path
4294                        param_exomiser_analysis_default_config_file = (
4295                            param_exomiser_preset
4296                        )
4297                    # elif os.path.exists(full_path(param_exomiser_preset)):
4298                    #     # Preset file is provided in full path
4299                    #     param_exomiser_analysis_default_config_file = full_path(param_exomiser_preset)
4300                    elif os.path.exists(
4301                        os.path.join(folder_config, param_exomiser_preset)
4302                    ):
4303                        # Preset file is provided a basename in config folder (can be a path with subfolders)
4304                        param_exomiser_analysis_default_config_file = os.path.join(
4305                            folder_config, param_exomiser_preset
4306                        )
4307                    else:
4308                        # Construct preset file
4309                        param_exomiser_analysis_default_config_file = os.path.join(
4310                            folder_config,
4311                            f"preset-{param_exomiser_preset}-analysis.json",
4312                        )
4313
4314                    # If preset file exists
4315                    param_exomiser_analysis_default_config_file = full_path(
4316                        param_exomiser_analysis_default_config_file
4317                    )
4318                    if os.path.exists(param_exomiser_analysis_default_config_file):
4319                        # Load prest file into analysis dict (either yaml or json)
4320                        with open(
4321                            param_exomiser_analysis_default_config_file
4322                        ) as json_file:
4323                            # param_exomiser_analysis_dict[""] = json.load(json_file)
4324                            param_exomiser_analysis_dict["analysis"] = yaml.safe_load(
4325                                json_file
4326                            )
4327
4328                    # Error preset file
4329                    else:
4330                        log.error(
4331                            f"No analysis preset config file ({param_exomiser_analysis_default_config_file})"
4332                        )
4333                        raise ValueError(
4334                            f"No analysis preset config file ({param_exomiser_analysis_default_config_file})"
4335                        )
4336
4337                # If no analysis dict created
4338                if not param_exomiser_analysis_dict:
4339                    log.error(f"No analysis config")
4340                    raise ValueError(f"No analysis config")
4341
4342                # Log
4343                log.debug(f"Pre analysis dict: {param_exomiser_analysis_dict}")
4344
4345                ### PHENOPACKET ###
4346                ###################
4347
4348                # If no PhenoPacket in analysis dict -> check in param
4349                if "phenopacket" not in param_exomiser_analysis_dict:
4350
4351                    # If PhenoPacket in param -> load anlaysis json
4352                    if param_exomiser.get("phenopacket", None):
4353
4354                        param_exomiser_phenopacket = param_exomiser.get("phenopacket")
4355                        param_exomiser_phenopacket = full_path(
4356                            param_exomiser_phenopacket
4357                        )
4358
4359                        # If param phenopacket is a file and exists
4360                        if isinstance(
4361                            param_exomiser_phenopacket, str
4362                        ) and os.path.exists(param_exomiser_phenopacket):
4363                            # Load phenopacket file into analysis dict (either yaml or json)
4364                            with open(param_exomiser_phenopacket) as json_file:
4365                                param_exomiser_analysis_dict["phenopacket"] = (
4366                                    yaml.safe_load(json_file)
4367                                )
4368
4369                        # If param phenopacket is a dict
4370                        elif isinstance(param_exomiser_phenopacket, dict):
4371                            # Load phenopacket dict into analysis dict (either yaml or json)
4372                            param_exomiser_analysis_dict["phenopacket"] = (
4373                                param_exomiser_phenopacket
4374                            )
4375
4376                        # Error phenopacket type
4377                        else:
4378                            log.error(f"Phenopacket type unknown. Check param file.")
4379                            raise ValueError(
4380                                f"Phenopacket type unknown. Check param file."
4381                            )
4382
4383                # If no PhenoPacket in analysis dict -> construct from sample and HPO in param
4384                if "phenopacket" not in param_exomiser_analysis_dict:
4385
4386                    # Init PhenoPacket
4387                    param_exomiser_analysis_dict["phenopacket"] = {
4388                        "id": "analysis",
4389                        "proband": {},
4390                    }
4391
4392                    ### Add subject ###
4393
4394                    # If subject exists
4395                    param_exomiser_subject = param_exomiser.get("subject", {})
4396
4397                    # If subject not exists -> found sample ID
4398                    if not param_exomiser_subject:
4399
4400                        # Found sample ID in param
4401                        sample = param_exomiser.get("sample", None)
4402
4403                        # Find sample ID (first sample)
4404                        if not sample:
4405                            sample_list = self.get_header_sample_list()
4406                            if len(sample_list) > 0:
4407                                sample = sample_list[0]
4408                            else:
4409                                log.error(f"No sample found")
4410                                raise ValueError(f"No sample found")
4411
4412                        # Create subject
4413                        param_exomiser_subject = {"id": sample, "sex": "UNKNOWN_SEX"}
4414
4415                    # Add to dict
4416                    param_exomiser_analysis_dict["phenopacket"][
4417                        "subject"
4418                    ] = param_exomiser_subject
4419
4420                    ### Add "phenotypicFeatures" ###
4421
4422                    # If phenotypicFeatures exists
4423                    param_exomiser_phenotypicfeatures = param_exomiser.get(
4424                        "phenotypicFeatures", []
4425                    )
4426
4427                    # If phenotypicFeatures not exists -> Try to infer from hpo list
4428                    if not param_exomiser_phenotypicfeatures:
4429
4430                        # Found HPO in param
4431                        param_exomiser_hpo = param_exomiser.get("hpo", [])
4432
4433                        # Split HPO if list in string format separated by comma
4434                        if isinstance(param_exomiser_hpo, str):
4435                            param_exomiser_hpo = param_exomiser_hpo.split(",")
4436
4437                        # Create HPO list
4438                        for hpo in param_exomiser_hpo:
4439                            hpo_clean = re.sub("[^0-9]", "", hpo)
4440                            param_exomiser_phenotypicfeatures.append(
4441                                {
4442                                    "type": {
4443                                        "id": f"HP:{hpo_clean}",
4444                                        "label": f"HP:{hpo_clean}",
4445                                    }
4446                                }
4447                            )
4448
4449                    # Add to dict
4450                    param_exomiser_analysis_dict["phenopacket"][
4451                        "phenotypicFeatures"
4452                    ] = param_exomiser_phenotypicfeatures
4453
4454                    # If phenotypicFeatures not exists -> Remove hiPhivePrioritiser step
4455                    if not param_exomiser_phenotypicfeatures:
4456                        for step in param_exomiser_analysis_dict.get(
4457                            "analysis", {}
4458                        ).get("steps", []):
4459                            if "hiPhivePrioritiser" in step:
4460                                param_exomiser_analysis_dict.get("analysis", {}).get(
4461                                    "steps", []
4462                                ).remove(step)
4463
4464                ### Add Input File ###
4465
4466                # Initial file name and htsFiles
4467                tmp_vcf_name = os.path.join(tmp_dir, "initial.vcf.gz")
4468                param_exomiser_analysis_dict["phenopacket"]["htsFiles"] = [
4469                    {
4470                        "uri": tmp_vcf_name,
4471                        "htsFormat": "VCF",
4472                        "genomeAssembly": assembly,
4473                    }
4474                ]
4475
4476                ### Add metaData ###
4477
4478                # If metaData not in analysis dict
4479                if "metaData" not in param_exomiser_analysis_dict:
4480                    param_exomiser_analysis_dict["phenopacket"]["metaData"] = {
4481                        "created": f"{datetime.datetime.now()}".replace(" ", "T") + "Z",
4482                        "createdBy": "howard",
4483                        "phenopacketSchemaVersion": 1,
4484                    }
4485
4486                ### OutputOptions ###
4487
4488                # Init output result folder
4489                output_results = os.path.join(tmp_dir, "results")
4490
4491                # If no outputOptions in analysis dict
4492                if "outputOptions" not in param_exomiser_analysis_dict:
4493
4494                    # default output formats
4495                    defaut_output_formats = ["TSV_VARIANT", "VCF"]
4496
4497                    # Get outputOptions in param
4498                    output_options = param_exomiser.get("outputOptions", None)
4499
4500                    # If no output_options in param -> check
4501                    if not output_options:
4502                        output_options = {
4503                            "outputContributingVariantsOnly": False,
4504                            "numGenes": 0,
4505                            "outputFormats": defaut_output_formats,
4506                        }
4507
4508                    # Replace outputDirectory in output options
4509                    output_options["outputDirectory"] = output_results
4510                    output_options["outputFileName"] = "howard"
4511
4512                    # Add outputOptions in analysis dict
4513                    param_exomiser_analysis_dict["outputOptions"] = output_options
4514
4515                else:
4516
4517                    # Replace output_results and output format (if exists in param)
4518                    param_exomiser_analysis_dict["outputOptions"][
4519                        "outputDirectory"
4520                    ] = output_results
4521                    param_exomiser_analysis_dict["outputOptions"]["outputFormats"] = (
4522                        list(
4523                            set(
4524                                param_exomiser_analysis_dict.get(
4525                                    "outputOptions", {}
4526                                ).get("outputFormats", [])
4527                                + ["TSV_VARIANT", "VCF"]
4528                            )
4529                        )
4530                    )
4531
4532                # log
4533                log.debug(f"Pre analysis dict: {param_exomiser_analysis_dict}")
4534
4535                ### ANALYSIS FILE ###
4536                #####################
4537
4538                ### Full JSON analysis config file ###
4539
4540                exomiser_analysis = os.path.join(tmp_dir, "analysis.json")
4541                with open(exomiser_analysis, "w") as fp:
4542                    json.dump(param_exomiser_analysis_dict, fp, indent=4)
4543
4544                ### SPLIT analysis and sample config files
4545
4546                # Splitted analysis dict
4547                param_exomiser_analysis_dict_for_split = (
4548                    param_exomiser_analysis_dict.copy()
4549                )
4550
4551                # Phenopacket JSON file
4552                exomiser_analysis_phenopacket = os.path.join(
4553                    tmp_dir, "analysis_phenopacket.json"
4554                )
4555                with open(exomiser_analysis_phenopacket, "w") as fp:
4556                    json.dump(
4557                        param_exomiser_analysis_dict_for_split.get("phenopacket"),
4558                        fp,
4559                        indent=4,
4560                    )
4561
4562                # Analysis JSON file without Phenopacket parameters
4563                param_exomiser_analysis_dict_for_split.pop("phenopacket")
4564                exomiser_analysis_analysis = os.path.join(
4565                    tmp_dir, "analysis_analysis.json"
4566                )
4567                with open(exomiser_analysis_analysis, "w") as fp:
4568                    json.dump(param_exomiser_analysis_dict_for_split, fp, indent=4)
4569
4570                ### INITAL VCF file ###
4571                #######################
4572
4573                ### Create list of samples to use and include inti initial VCF file ####
4574
4575                # Subject (main sample)
4576                # Get sample ID in analysis dict
4577                sample_subject = (
4578                    param_exomiser_analysis_dict.get("phenopacket", {})
4579                    .get("subject", {})
4580                    .get("id", None)
4581                )
4582                sample_proband = (
4583                    param_exomiser_analysis_dict.get("phenopacket", {})
4584                    .get("proband", {})
4585                    .get("subject", {})
4586                    .get("id", None)
4587                )
4588                sample = []
4589                if sample_subject:
4590                    sample.append(sample_subject)
4591                if sample_proband:
4592                    sample.append(sample_proband)
4593
4594                # Get sample ID within Pedigree
4595                pedigree_persons_list = (
4596                    param_exomiser_analysis_dict.get("phenopacket", {})
4597                    .get("pedigree", {})
4598                    .get("persons", {})
4599                )
4600
4601                # Create list with all sample ID in pedigree (if exists)
4602                pedigree_persons = []
4603                for person in pedigree_persons_list:
4604                    pedigree_persons.append(person.get("individualId"))
4605
4606                # Concat subject sample ID and samples ID in pedigreesamples
4607                samples = list(set(sample + pedigree_persons))
4608
4609                # Check if sample list is not empty
4610                if not samples:
4611                    log.error(f"No samples found")
4612                    raise ValueError(f"No samples found")
4613
4614                # Create VCF with sample (either sample in param or first one by default)
4615                # Export VCF file
4616                self.export_variant_vcf(
4617                    vcf_file=tmp_vcf_name,
4618                    remove_info=True,
4619                    add_samples=True,
4620                    list_samples=samples,
4621                    index=False,
4622                )
4623
4624                ### Execute Exomiser ###
4625                ########################
4626
4627                # Init command
4628                exomiser_command = ""
4629
4630                # Command exomiser options
4631                exomiser_options = f" --spring.config.location={databases_folders}/{assembly}/application.properties --exomiser.data-directory={databases_folders}/{assembly} "
4632
4633                # Release
4634                exomiser_release = param_exomiser.get("release", None)
4635                if exomiser_release:
4636                    # phenotype data version
4637                    exomiser_options += (
4638                        f" --exomiser.phenotype.data-version={exomiser_release} "
4639                    )
4640                    # data version
4641                    exomiser_options += (
4642                        f" --exomiser.{assembly}.data-version={exomiser_release} "
4643                    )
4644                    # variant white list
4645                    variant_white_list_file = (
4646                        f"{exomiser_release}_{assembly}_clinvar_whitelist.tsv.gz"
4647                    )
4648                    if os.path.exists(
4649                        os.path.join(
4650                            databases_folders, assembly, variant_white_list_file
4651                        )
4652                    ):
4653                        exomiser_options += f" --exomiser.{assembly}.variant-white-list-path={variant_white_list_file} "
4654
4655                # transcript_source
4656                transcript_source = param_exomiser.get(
4657                    "transcript_source", None
4658                )  # ucsc, refseq, ensembl
4659                if transcript_source:
4660                    exomiser_options += (
4661                        f" --exomiser.{assembly}.transcript-source={transcript_source} "
4662                    )
4663
4664                # If analysis contain proband param
4665                if param_exomiser_analysis_dict.get("phenopacket", {}).get(
4666                    "proband", {}
4667                ):
4668                    exomiser_command_analysis = f" {exomiser_bin_command} --analysis={exomiser_analysis_analysis} --sample={exomiser_analysis_phenopacket} {exomiser_options} "
4669
4670                # If no proband (usually uniq sample)
4671                else:
4672                    exomiser_command_analysis = f" {exomiser_bin_command} --analysis={exomiser_analysis} {exomiser_options}"
4673
4674                # Log
4675                log.debug(f"exomiser_command_analysis={exomiser_command_analysis}")
4676
4677                # Run command
4678                result = subprocess.call(
4679                    exomiser_command_analysis.split(), stdout=subprocess.PIPE
4680                )
4681                if result:
4682                    log.error("Exomiser command failed")
4683                    raise ValueError("Exomiser command failed")
4684
4685                ### RESULTS ###
4686                ###############
4687
4688                ### Annotate with TSV fields ###
4689
4690                # Init result tsv file
4691                exomiser_to_info = param_exomiser.get("exomiser_to_info", False)
4692
4693                # Init result tsv file
4694                output_results_tsv = os.path.join(output_results, "howard.variants.tsv")
4695
4696                # Parse TSV file and explode columns in INFO field
4697                if exomiser_to_info and os.path.exists(output_results_tsv):
4698
4699                    # Log
4700                    log.debug("Exomiser columns to VCF INFO field")
4701
4702                    # Retrieve columns and types
4703                    query = f""" SELECT * FROM read_csv('{output_results_tsv}', auto_detect=True, delim='\t', sample_size=-1) LIMIT 0 """
4704                    output_results_tsv_df = self.get_query_to_df(query)
4705                    output_results_tsv_columns = output_results_tsv_df.columns.tolist()
4706
4707                    # Init concat fields for update
4708                    sql_query_update_concat_fields = []
4709
4710                    # Fields to avoid
4711                    fields_to_avoid = [
4712                        "CONTIG",
4713                        "START",
4714                        "END",
4715                        "REF",
4716                        "ALT",
4717                        "QUAL",
4718                        "FILTER",
4719                        "GENOTYPE",
4720                    ]
4721
4722                    # List all columns to add into header
4723                    for header_column in output_results_tsv_columns:
4724
4725                        # If header column is enable
4726                        if header_column not in fields_to_avoid:
4727
4728                            # Header info type
4729                            header_info_type = "String"
4730                            header_column_df = output_results_tsv_df[header_column]
4731                            header_column_df_dtype = header_column_df.dtype
4732                            if header_column_df_dtype == object:
4733                                if (
4734                                    pd.to_numeric(header_column_df, errors="coerce")
4735                                    .notnull()
4736                                    .all()
4737                                ):
4738                                    header_info_type = "Float"
4739                            else:
4740                                header_info_type = "Integer"
4741
4742                            # Header info
4743                            characters_to_validate = ["-"]
4744                            pattern = "[" + "".join(characters_to_validate) + "]"
4745                            header_info_name = re.sub(
4746                                pattern,
4747                                "_",
4748                                f"Exomiser_{header_column}".replace("#", ""),
4749                            )
4750                            header_info_number = "."
4751                            header_info_description = (
4752                                f"Exomiser {header_column} annotation"
4753                            )
4754                            header_info_source = "Exomiser"
4755                            header_info_version = "unknown"
4756                            header_info_code = CODE_TYPE_MAP[header_info_type]
4757                            vcf_reader.infos[header_info_name] = vcf.parser._Info(
4758                                header_info_name,
4759                                header_info_number,
4760                                header_info_type,
4761                                header_info_description,
4762                                header_info_source,
4763                                header_info_version,
4764                                header_info_code,
4765                            )
4766
4767                            # Add field to add for update to concat fields
4768                            sql_query_update_concat_fields.append(
4769                                f"""
4770                                CASE
4771                                    WHEN table_parquet."{header_column}" NOT IN ('','.')
4772                                    THEN concat(
4773                                        '{header_info_name}=',
4774                                        table_parquet."{header_column}",
4775                                        ';'
4776                                        )
4777
4778                                    ELSE ''
4779                                END
4780                            """
4781                            )
4782
4783                    # Update query
4784                    sql_query_update = f"""
4785                        UPDATE {table_variants} as table_variants
4786                            SET INFO = concat(
4787                                            CASE
4788                                                WHEN INFO NOT IN ('', '.')
4789                                                THEN INFO
4790                                                ELSE ''
4791                                            END,
4792                                            CASE
4793                                                WHEN table_variants.INFO NOT IN ('','.')
4794                                                THEN ';'
4795                                                ELSE ''
4796                                            END,
4797                                            (
4798                                            SELECT 
4799                                                concat(
4800                                                    {",".join(sql_query_update_concat_fields)}
4801                                                )
4802                                            FROM read_csv('{output_results_tsv}', auto_detect=True, delim='\t', sample_size=-1) as table_parquet
4803                                                    WHERE concat('chr', CAST(table_parquet.\"CONTIG\" AS STRING)) = table_variants.\"#CHROM\"
4804                                                    AND table_parquet.\"START\" = table_variants.\"POS\"
4805                                                    AND table_parquet.\"ALT\" = table_variants.\"ALT\"
4806                                                    AND table_parquet.\"REF\" = table_variants.\"REF\"
4807                                            )
4808                                        )
4809                            ;
4810                        """
4811
4812                    # Update
4813                    self.conn.execute(sql_query_update)
4814
4815                ### Annotate with VCF INFO field ###
4816
4817                # Init result VCF file
4818                output_results_vcf = os.path.join(output_results, "howard.vcf.gz")
4819
4820                # If VCF exists
4821                if os.path.exists(output_results_vcf):
4822
4823                    # Log
4824                    log.debug("Exomiser result VCF update variants")
4825
4826                    # Find Exomiser INFO field annotation in header
4827                    with gzip.open(output_results_vcf, "rt") as f:
4828                        header_list = self.read_vcf_header(f)
4829                    exomiser_vcf_header = vcf.Reader(
4830                        io.StringIO("\n".join(header_list))
4831                    )
4832
4833                    # Add annotation INFO field to header
4834                    vcf_reader.infos["Exomiser"] = exomiser_vcf_header.infos["Exomiser"]
4835
4836                    # Update variants with VCF
4837                    self.update_from_vcf(output_results_vcf)
4838
4839        return True

This function annotate with Exomiser

This function uses args as parameters, in section "annotation" -> "exomiser", with sections:

"analysis" (dict/file): Full analysis dictionnary parameters (see Exomiser docs). Either a dict, or a file in JSON or YAML format. These parameters may change depending on other parameters (e.g. phenotipicFeatures/HPO) Default : None
"preset" (string): Analysis preset (available in config folder). Used if no full "analysis" is provided. Default: "exome"
"phenopacket" (dict/file): Samples and phenotipic features parameters (see Exomiser docs). Either a dict, or a file in JSON or YAML format. Default: None
"subject" (dict): Sample parameters (see Exomiser docs). Example: "subject": { "id": "ISDBM322017", "sex": "FEMALE" } Default: None
"sample" (string): Sample name to construct "subject" section: "subject": { "id": "", "sex": "UNKNOWN_SEX" } Default: None
"phenotypicFeatures" (dict) Phenotypic features to construct "subject" section. Example: "phenotypicFeatures": [ { "type": { "id": "HP:0001159", "label": "Syndactyly" } }, { "type": { "id": "HP:0000486", "label": "Strabismus" } } ]
"hpo" (list) List of HPO ids as phenotypic features. Example: "hpo": ['0001156', '0001363', '0011304', '0010055'] Default: []
"outputOptions" (dict): Output options (see Exomiser docs). Default: "output_options" = { "outputContributingVariantsOnly": False, "numGenes": 0, "outputFormats": ["TSV_VARIANT", "VCF"] }
"transcript_source" (string): Transcript source (either "refseq", "ucsc", "ensembl") Default: "refseq"
"exomiser_to_info" (boolean): Add exomiser TSV file columns as INFO fields in VCF. Default: False
"release" (string): Exomise database release. If not exists, database release will be downloaded (take a while). Default: None (provided by application.properties configuration file)
"exomiser_application_properties" (file): Exomiser configuration file (see Exomiser docs). Useful to automatically download databases (especially for specific genome databases).

Notes:

If no sample in parameters, first sample in VCF will be chosen
If no HPO found, "hiPhivePrioritiser" analysis step will be switch off

Parameters

threads: The number of threads to use

Returns

None.

def annotation_snpeff(self, threads: int = None) -> None: View Source

4841    def annotation_snpeff(self, threads: int = None) -> None:
4842        """
4843        This function annotate with snpEff
4844
4845        :param threads: The number of threads to use
4846        :return: the value of the variable "return_value".
4847        """
4848
4849        # DEBUG
4850        log.debug("Start annotation with snpeff databases")
4851
4852        # Threads
4853        if not threads:
4854            threads = self.get_threads()
4855        log.debug("Threads: " + str(threads))
4856
4857        # DEBUG
4858        delete_tmp = True
4859        if self.get_config().get("verbosity", "warning") in ["debug"]:
4860            delete_tmp = False
4861            log.debug("Delete tmp files/folders: " + str(delete_tmp))
4862
4863        # Config
4864        config = self.get_config()
4865        log.debug("Config: " + str(config))
4866
4867        # Config - Folders - Databases
4868        databases_folders = (
4869            config.get("folders", {}).get("databases", {}).get("snpeff", ["."])
4870        )
4871        log.debug("Databases annotations: " + str(databases_folders))
4872
4873        # # Config - Java
4874        # java_bin = get_bin(
4875        #     tool="java",
4876        #     bin="java",
4877        #     bin_type="bin",
4878        #     config=config,
4879        #     default_folder="/usr/bin",
4880        # )
4881        # if not (os.path.exists(java_bin) or (java_bin and which(java_bin))):
4882        #     log.error(f"Annotation failed: no java bin '{java_bin}'")
4883        #     raise ValueError(f"Annotation failed: no java bin '{java_bin}'")
4884
4885        # # Config - snpEff bin
4886        # snpeff_jar = get_bin(
4887        #     tool="snpeff",
4888        #     bin="snpEff.jar",
4889        #     bin_type="jar",
4890        #     config=config,
4891        #     default_folder=f"{DEFAULT_TOOLS_FOLDER}/snpeff",
4892        # )
4893        # if not (os.path.exists(snpeff_jar) or (snpeff_jar and which(snpeff_jar))):
4894        #     log.error(f"Annotation failed: no snpEff jar '{snpeff_jar}'")
4895        #     raise ValueError(f"Annotation failed: no snpEff jar '{snpeff_jar}'")
4896
4897        # Config - snpEff bin command
4898        snpeff_bin_command = get_bin_command(
4899            bin="snpEff.jar",
4900            tool="snpeff",
4901            bin_type="jar",
4902            config=config,
4903            default_folder=f"{DEFAULT_TOOLS_FOLDER}/snpeff",
4904        )
4905        if not snpeff_bin_command:
4906            msg_err = f"Annotation failed: no snpeff bin '{snpeff_bin_command}'"
4907            log.error(msg_err)
4908            raise ValueError(msg_err)
4909
4910        # Config - snpEff databases
4911        snpeff_databases = (
4912            config.get("folders", {})
4913            .get("databases", {})
4914            .get("snpeff", DEFAULT_SNPEFF_FOLDER)
4915        )
4916        snpeff_databases = full_path(snpeff_databases)
4917        if snpeff_databases is not None and snpeff_databases != "":
4918            log.debug(f"Create snpEff databases folder")
4919            if not os.path.exists(snpeff_databases):
4920                os.makedirs(snpeff_databases)
4921
4922        # Param
4923        param = self.get_param()
4924        log.debug("Param: " + str(param))
4925
4926        # Param
4927        options = param.get("annotation", {}).get("snpeff", {}).get("options", None)
4928        log.debug("Options: " + str(options))
4929
4930        # Param - Assembly
4931        assembly = param.get("assembly", config.get("assembly", DEFAULT_ASSEMBLY))
4932
4933        # Param - Options
4934        snpeff_options = (
4935            param.get("annotation", {}).get("snpeff", {}).get("options", "")
4936        )
4937        snpeff_stats = param.get("annotation", {}).get("snpeff", {}).get("stats", None)
4938        snpeff_csvstats = (
4939            param.get("annotation", {}).get("snpeff", {}).get("csvStats", None)
4940        )
4941        if snpeff_stats:
4942            snpeff_stats = snpeff_stats.replace("OUTPUT", self.get_output())
4943            snpeff_stats = full_path(snpeff_stats)
4944            snpeff_options += f" -stats {snpeff_stats}"
4945        if snpeff_csvstats:
4946            snpeff_csvstats = snpeff_csvstats.replace("OUTPUT", self.get_output())
4947            snpeff_csvstats = full_path(snpeff_csvstats)
4948            snpeff_options += f" -csvStats {snpeff_csvstats}"
4949
4950        # Data
4951        table_variants = self.get_table_variants()
4952
4953        # Check if not empty
4954        log.debug("Check if not empty")
4955        sql_query_chromosomes = (
4956            f"""SELECT count(*) as count FROM {table_variants} as table_variants"""
4957        )
4958        # if not self.conn.execute(f"{sql_query_chromosomes}").df()["count"][0]:
4959        if not self.get_query_to_df(f"{sql_query_chromosomes}")["count"][0]:
4960            log.info(f"VCF empty")
4961            return
4962
4963        # Export in VCF
4964        log.debug("Create initial file to annotate")
4965        tmp_vcf = NamedTemporaryFile(
4966            prefix=self.get_prefix(),
4967            dir=self.get_tmp_dir(),
4968            suffix=".vcf.gz",
4969            delete=True,
4970        )
4971        tmp_vcf_name = tmp_vcf.name
4972
4973        # VCF header
4974        vcf_reader = self.get_header()
4975        log.debug("Initial header: " + str(vcf_reader.infos))
4976
4977        # Existing annotations
4978        for vcf_annotation in self.get_header().infos:
4979
4980            vcf_annotation_line = self.get_header().infos.get(vcf_annotation)
4981            log.debug(
4982                f"Existing annotations in VCF: {vcf_annotation} [{vcf_annotation_line}]"
4983            )
4984
4985        # Memory limit
4986        # if config.get("memory", None):
4987        #     memory_limit = config.get("memory", "8G")
4988        # else:
4989        #     memory_limit = "8G"
4990        memory_limit = self.get_memory("8G")
4991        log.debug(f"memory_limit: {memory_limit}")
4992
4993        # snpEff java options
4994        snpeff_java_options = (
4995            f" -Xmx{memory_limit} -XX:+UseParallelGC -XX:ParallelGCThreads={threads} "
4996        )
4997        log.debug(f"Exomiser java options: {snpeff_java_options}")
4998
4999        force_update_annotation = True
5000
5001        if "ANN" not in self.get_header().infos or force_update_annotation:
5002
5003            # Check snpEff database
5004            log.debug(f"Check snpEff databases {[assembly]}")
5005            databases_download_snpeff(
5006                folder=snpeff_databases, assemblies=[assembly], config=config
5007            )
5008
5009            # Export VCF file
5010            self.export_variant_vcf(
5011                vcf_file=tmp_vcf_name,
5012                remove_info=True,
5013                add_samples=False,
5014                index=True,
5015            )
5016
5017            # Tmp file
5018            err_files = []
5019            tmp_annotate_vcf = NamedTemporaryFile(
5020                prefix=self.get_prefix(),
5021                dir=self.get_tmp_dir(),
5022                suffix=".vcf",
5023                delete=False,
5024            )
5025            tmp_annotate_vcf_name = tmp_annotate_vcf.name
5026            tmp_annotate_vcf_name_err = tmp_annotate_vcf_name + ".err"
5027            err_files.append(tmp_annotate_vcf_name_err)
5028
5029            # Command
5030            snpeff_command = f"{snpeff_bin_command} {assembly} -dataDir {snpeff_databases} {snpeff_options} {tmp_vcf_name} 1>{tmp_annotate_vcf_name} 2>>{tmp_annotate_vcf_name_err}"
5031            log.debug(f"Annotation - snpEff command: {snpeff_command}")
5032            run_parallel_commands([snpeff_command], 1)
5033
5034            # Error messages
5035            log.info(f"Error/Warning messages:")
5036            error_message_command_all = []
5037            error_message_command_warning = []
5038            error_message_command_err = []
5039            for err_file in err_files:
5040                with open(err_file, "r") as f:
5041                    for line in f:
5042                        message = line.strip()
5043                        error_message_command_all.append(message)
5044                        if line.startswith("[W::"):
5045                            error_message_command_warning.append(message)
5046                        if line.startswith("[E::"):
5047                            error_message_command_err.append(f"{err_file}: " + message)
5048            # log info
5049            for message in list(
5050                set(error_message_command_err + error_message_command_warning)
5051            ):
5052                log.info(f"   {message}")
5053            # debug info
5054            for message in list(set(error_message_command_all)):
5055                log.debug(f"   {message}")
5056            # failed
5057            if len(error_message_command_err):
5058                log.error("Annotation failed: Error in commands")
5059                raise ValueError("Annotation failed: Error in commands")
5060
5061            # Find annotation in header
5062            with open(tmp_annotate_vcf_name, "rt") as f:
5063                header_list = self.read_vcf_header(f)
5064            annovar_vcf_header = vcf.Reader(io.StringIO("\n".join(header_list)))
5065
5066            for ann in annovar_vcf_header.infos:
5067                if ann not in self.get_header().infos:
5068                    vcf_reader.infos[ann] = annovar_vcf_header.infos.get(ann)
5069
5070            # Update variants
5071            log.info(f"Annotation - Updating...")
5072            self.update_from_vcf(tmp_annotate_vcf_name)
5073
5074        else:
5075            if "ANN" in self.get_header().infos:
5076                log.debug(f"Existing snpEff annotations in VCF")
5077            if force_update_annotation:
5078                log.debug(f"Existing snpEff annotations in VCF - annotation forced")

This function annotate with snpEff

Parameters

threads: The number of threads to use

Returns

the value of the variable "return_value".

def annotation_annovar(self, threads: int = None) -> None: View Source

5080    def annotation_annovar(self, threads: int = None) -> None:
5081        """
5082        It takes a VCF file, annotates it with Annovar, and then updates the database with the new
5083        annotations
5084
5085        :param threads: number of threads to use
5086        :return: the value of the variable "return_value".
5087        """
5088
5089        # DEBUG
5090        log.debug("Start annotation with Annovar databases")
5091
5092        # Threads
5093        if not threads:
5094            threads = self.get_threads()
5095        log.debug("Threads: " + str(threads))
5096
5097        # Tmp en Err files
5098        tmp_files = []
5099        err_files = []
5100
5101        # DEBUG
5102        delete_tmp = True
5103        if self.get_config().get("verbosity", "warning") in ["debug"]:
5104            delete_tmp = False
5105            log.debug("Delete tmp files/folders: " + str(delete_tmp))
5106
5107        # Config
5108        config = self.get_config()
5109        log.debug("Config: " + str(config))
5110
5111        # Config - Folders - Databases
5112        databases_folders = (
5113            config.get("folders", {}).get("databases", {}).get("annovar", ["."])
5114        )
5115        log.debug("Databases annotations: " + str(databases_folders))
5116
5117        # Config - annovar bin command
5118        annovar_bin_command = get_bin_command(
5119            bin="table_annovar.pl",
5120            tool="annovar",
5121            bin_type="perl",
5122            config=config,
5123            default_folder=f"{DEFAULT_TOOLS_FOLDER}/annovar",
5124        )
5125        if not annovar_bin_command:
5126            msg_err = f"Annotation failed: no annovar bin '{annovar_bin_command}'"
5127            log.error(msg_err)
5128            raise ValueError(msg_err)
5129
5130        # Config - BCFTools bin command
5131        bcftools_bin_command = get_bin_command(
5132            bin="bcftools",
5133            tool="bcftools",
5134            bin_type="bin",
5135            config=config,
5136            default_folder=f"{DEFAULT_TOOLS_FOLDER}/bcftools",
5137        )
5138        if not bcftools_bin_command:
5139            msg_err = f"Annotation failed: no bcftools bin '{bcftools_bin_command}'"
5140            log.error(msg_err)
5141            raise ValueError(msg_err)
5142
5143        # Config - annovar databases
5144        annovar_databases = (
5145            config.get("folders", {})
5146            .get("databases", {})
5147            .get("annovar", DEFAULT_ANNOVAR_FOLDER)
5148        )
5149        annovar_databases = full_path(annovar_databases)
5150        if annovar_databases != "" and not os.path.exists(annovar_databases):
5151            os.makedirs(annovar_databases)
5152
5153        # Param
5154        param = self.get_param()
5155        log.debug("Param: " + str(param))
5156
5157        # Param - options
5158        options = param.get("annotation", {}).get("annovar", {}).get("options", {})
5159        log.debug("Options: " + str(options))
5160
5161        # Param - annotations
5162        annotations = (
5163            param.get("annotation", {}).get("annovar", {}).get("annotations", {})
5164        )
5165        log.debug("Annotations: " + str(annotations))
5166
5167        # Param - Assembly
5168        assembly = param.get("assembly", config.get("assembly", DEFAULT_ASSEMBLY))
5169
5170        # Annovar database assembly
5171        annovar_databases_assembly = f"{annovar_databases}/{assembly}"
5172        if annovar_databases_assembly != "" and not os.path.exists(
5173            annovar_databases_assembly
5174        ):
5175            os.makedirs(annovar_databases_assembly)
5176
5177        # Data
5178        table_variants = self.get_table_variants()
5179
5180        # Check if not empty
5181        log.debug("Check if not empty")
5182        sql_query_chromosomes = (
5183            f"""SELECT count(*) as count FROM {table_variants} as table_variants"""
5184        )
5185        sql_query_chromosomes_df = self.get_query_to_df(sql_query_chromosomes)
5186        if not sql_query_chromosomes_df["count"][0]:
5187            log.info(f"VCF empty")
5188            return
5189
5190        # VCF header
5191        vcf_reader = self.get_header()
5192        log.debug("Initial header: " + str(vcf_reader.infos))
5193
5194        # Existing annotations
5195        for vcf_annotation in self.get_header().infos:
5196
5197            vcf_annotation_line = self.get_header().infos.get(vcf_annotation)
5198            log.debug(
5199                f"Existing annotations in VCF: {vcf_annotation} [{vcf_annotation_line}]"
5200            )
5201
5202        force_update_annotation = True
5203
5204        if annotations:
5205
5206            commands = []
5207            tmp_annotates_vcf_name_list = []
5208
5209            # Export in VCF
5210            log.debug("Create initial file to annotate")
5211            tmp_vcf = NamedTemporaryFile(
5212                prefix=self.get_prefix(),
5213                dir=self.get_tmp_dir(),
5214                suffix=".vcf.gz",
5215                delete=False,
5216            )
5217            tmp_vcf_name = tmp_vcf.name
5218            tmp_files.append(tmp_vcf_name)
5219            tmp_files.append(tmp_vcf_name + ".tbi")
5220
5221            # Export VCF file
5222            self.export_variant_vcf(
5223                vcf_file=tmp_vcf_name,
5224                remove_info=".",
5225                add_samples=False,
5226                index=True,
5227            )
5228
5229            # Create file for field rename
5230            log.debug("Create file for field rename")
5231            tmp_rename = NamedTemporaryFile(
5232                prefix=self.get_prefix(),
5233                dir=self.get_tmp_dir(),
5234                suffix=".rename",
5235                delete=False,
5236            )
5237            tmp_rename_name = tmp_rename.name
5238            tmp_files.append(tmp_rename_name)
5239
5240            # Check Annovar database
5241            log.debug(
5242                f"Check Annovar databases {[assembly]}: {list(annotations.keys())}"
5243            )
5244            databases_download_annovar(
5245                folder=annovar_databases,
5246                files=list(annotations.keys()),
5247                assemblies=[assembly],
5248            )
5249
5250            for annotation in annotations:
5251                annotation_fields = annotations[annotation]
5252
5253                if not annotation_fields:
5254                    annotation_fields = {"INFO": None}
5255
5256                log.info(f"Annotations Annovar - database '{annotation}'")
5257                log.debug(f"Annotation '{annotation}' - fields: {annotation_fields}")
5258
5259                # Tmp file for annovar
5260                err_files = []
5261                tmp_annotate_vcf_directory = TemporaryDirectory(
5262                    prefix=self.get_prefix(), dir=self.get_tmp_dir(), suffix=".annovar"
5263                )
5264                tmp_annotate_vcf_prefix = tmp_annotate_vcf_directory.name + "/annovar"
5265                tmp_annotate_vcf_name_annovar = (
5266                    tmp_annotate_vcf_prefix + "." + assembly + "_multianno.vcf"
5267                )
5268                tmp_annotate_vcf_name_err = tmp_annotate_vcf_directory.name + "/.err"
5269                err_files.append(tmp_annotate_vcf_name_err)
5270                tmp_files.append(tmp_annotate_vcf_name_err)
5271
5272                # Tmp file final vcf annotated by annovar
5273                tmp_annotate_vcf = NamedTemporaryFile(
5274                    prefix=self.get_prefix(),
5275                    dir=self.get_tmp_dir(),
5276                    suffix=".vcf.gz",
5277                    delete=False,
5278                )
5279                tmp_annotate_vcf_name = tmp_annotate_vcf.name
5280                tmp_annotates_vcf_name_list.append(tmp_annotate_vcf_name)
5281                tmp_files.append(tmp_annotate_vcf_name)
5282                tmp_files.append(tmp_annotate_vcf_name + ".tbi")
5283
5284                # Number of fields
5285                annotation_list = []
5286                annotation_renamed_list = []
5287
5288                for annotation_field in annotation_fields:
5289
5290                    # field new name, if parametered SKIPPED !!!!!! not managed actually TODO
5291                    annotation_fields_new_name = annotation_fields.get(
5292                        annotation_field, annotation_field
5293                    )
5294                    if not annotation_fields_new_name:
5295                        annotation_fields_new_name = annotation_field
5296
5297                    if (
5298                        force_update_annotation
5299                        or annotation_fields_new_name not in self.get_header().infos
5300                    ):
5301                        annotation_list.append(annotation_field)
5302                        annotation_renamed_list.append(annotation_fields_new_name)
5303                    else:  # annotation_fields_new_name in self.get_header().infos and not force_update_annotation:
5304                        log.warning(
5305                            f"Annotation '{annotation}' - '{annotation_fields_new_name}' - already exists (skipped)"
5306                        )
5307
5308                    # Add rename info
5309                    run_parallel_commands(
5310                        [
5311                            f"echo 'INFO/{annotation_field} {annotation_fields_new_name}' >> {tmp_rename_name}"
5312                        ],
5313                        1,
5314                    )
5315
5316                # log.debug("fields_to_removed: " + str(fields_to_removed))
5317                log.debug("annotation_list: " + str(annotation_list))
5318
5319                # protocol
5320                protocol = annotation
5321
5322                # argument
5323                argument = ""
5324
5325                # operation
5326                operation = "f"
5327                if annotation in ["refGene", "refGeneWithVer"] or annotation.startswith(
5328                    "ensGene"
5329                ):
5330                    operation = "g"
5331                    if options.get("genebase", None):
5332                        argument = f"""'{options.get("genebase","")}'"""
5333                elif annotation in ["cytoBand"]:
5334                    operation = "r"
5335
5336                # argument option
5337                argument_option = ""
5338                if argument != "":
5339                    argument_option = " --argument " + argument
5340
5341                # command options
5342                command_options = f""" --nastring . --vcfinput --polish --dot2underline --thread {threads} """  # --intronhgvs 10
5343                for option in options:
5344                    if option not in ["genebase"]:
5345                        command_options += f""" --{option}={options[option]}"""
5346
5347                # Command
5348
5349                # Command - Annovar
5350                command_annovar = f"""{annovar_bin_command} {tmp_vcf_name} {annovar_databases_assembly} --buildver {assembly} --outfile {tmp_annotate_vcf_prefix} --remove --protocol {protocol} --operation {operation} {argument_option} {command_options} 2>>{tmp_annotate_vcf_name_err} && mv {tmp_annotate_vcf_name_annovar} {tmp_annotate_vcf_name}.tmp.vcf """
5351                tmp_files.append(f"{tmp_annotate_vcf_name}.tmp.vcf")
5352
5353                # Command - start pipe
5354                command_annovar += f""" && {bcftools_bin_command} view --threads={threads} {tmp_annotate_vcf_name}.tmp.vcf 2>>{tmp_annotate_vcf_name_err} """
5355
5356                # Command - Clean INFO/ANNOVAR_DATE (due to Annovar issue with multiple TAGS!)
5357                command_annovar += """ | sed "s/ANNOVAR_DATE=[^;\t]*;//gi" """
5358
5359                # Command - Special characters (refGene annotation)
5360                command_annovar += """ | sed "s/\\\\\\x3b/,/gi" """
5361
5362                # Command - Clean empty fields (with value ".")
5363                command_annovar += """ | awk -F'\\t' -v OFS='\\t' '{if ($0 ~ /^#/) print; else {split($8,a,";");for(i=1;i<=length(a);i++) {split(a[i],b,"=");if(b[2]!=".") {c[b[1]]=b[2]}}; split($8,d,";");for(i=1;i<=length(d);i++) {split(d[i],e,"=");if(c[e[1]]!="") {if(info!="") {info=info";"}; info=info""e[1]"="c[e[1]]}}; if(info!="") {$8=info} else {$8=""}; delete c; info=""; print}}' """
5364
5365                # Command - Extract only needed fields, and remove ANNOVAR fields, and compress and index final file
5366                annovar_fields_to_keep = ["INFO/ANNOVAR_DATE", "INFO/ALLELE_END"]
5367                if "ALL" not in annotation_list and "INFO" not in annotation_list:
5368                    # for ann in annotation_renamed_list:
5369                    for ann in annotation_list:
5370                        annovar_fields_to_keep.append(f"^INFO/{ann}")
5371
5372                command_annovar += f""" | {bcftools_bin_command} annotate --pair-logic exact --threads={threads} -x {",".join(annovar_fields_to_keep)} --rename-annots={tmp_rename_name} -o {tmp_annotate_vcf_name} -Oz 2>>{tmp_annotate_vcf_name_err} """
5373
5374                # Command - indexing
5375                command_annovar += f"""  && tabix {tmp_annotate_vcf_name} """
5376
5377                log.debug(f"Annotation - Annovar command: {command_annovar}")
5378                run_parallel_commands([command_annovar], 1)
5379
5380                # Error messages
5381                log.info(f"Error/Warning messages:")
5382                error_message_command_all = []
5383                error_message_command_warning = []
5384                error_message_command_err = []
5385                for err_file in err_files:
5386                    with open(err_file, "r") as f:
5387                        for line in f:
5388                            message = line.strip()
5389                            error_message_command_all.append(message)
5390                            if line.startswith("[W::") or line.startswith("WARNING"):
5391                                error_message_command_warning.append(message)
5392                            if line.startswith("[E::") or line.startswith("ERROR"):
5393                                error_message_command_err.append(
5394                                    f"{err_file}: " + message
5395                                )
5396                # log info
5397                for message in list(
5398                    set(error_message_command_err + error_message_command_warning)
5399                ):
5400                    log.info(f"   {message}")
5401                # debug info
5402                for message in list(set(error_message_command_all)):
5403                    log.debug(f"   {message}")
5404                # failed
5405                if len(error_message_command_err):
5406                    log.error("Annotation failed: Error in commands")
5407                    raise ValueError("Annotation failed: Error in commands")
5408
5409            if tmp_annotates_vcf_name_list:
5410
5411                # List of annotated files
5412                tmp_annotates_vcf_name_to_merge = " ".join(tmp_annotates_vcf_name_list)
5413
5414                # Tmp file
5415                tmp_annotate_vcf = NamedTemporaryFile(
5416                    prefix=self.get_prefix(),
5417                    dir=self.get_tmp_dir(),
5418                    suffix=".vcf.gz",
5419                    delete=False,
5420                )
5421                tmp_annotate_vcf_name = tmp_annotate_vcf.name
5422                tmp_files.append(tmp_annotate_vcf_name)
5423                tmp_annotate_vcf_name_err = tmp_annotate_vcf_name + ".err"
5424                err_files.append(tmp_annotate_vcf_name_err)
5425                tmp_files.append(tmp_annotate_vcf_name_err)
5426
5427                # Command merge
5428                merge_command = f"{bcftools_bin_command} merge --force-samples --threads={threads} {tmp_vcf_name} {tmp_annotates_vcf_name_to_merge} -o {tmp_annotate_vcf_name} -Oz 2>>{tmp_annotate_vcf_name_err} "
5429                log.info(
5430                    f"Annotation Annovar - Annotation merging "
5431                    + str(len(tmp_annotates_vcf_name_list))
5432                    + " annotated files"
5433                )
5434                log.debug(f"Annotation - merge command: {merge_command}")
5435                run_parallel_commands([merge_command], 1)
5436
5437                # Find annotation in header
5438                with bgzf.open(tmp_annotate_vcf_name, "rt") as f:
5439                    header_list = self.read_vcf_header(f)
5440                annovar_vcf_header = vcf.Reader(io.StringIO("\n".join(header_list)))
5441
5442                for ann in annovar_vcf_header.infos:
5443                    if ann not in self.get_header().infos:
5444                        vcf_reader.infos[ann] = annovar_vcf_header.infos.get(ann)
5445
5446                # Update variants
5447                log.info(f"Annotation Annovar - Updating...")
5448                self.update_from_vcf(tmp_annotate_vcf_name)
5449
5450            # Clean files
5451            # Tmp file remove command
5452            if True:
5453                tmp_files_remove_command = ""
5454                if tmp_files:
5455                    tmp_files_remove_command = " ".join(tmp_files)
5456                clean_command = f" rm -f {tmp_files_remove_command} "
5457                log.debug(f"Annotation Annovar - Annotation cleaning ")
5458                log.debug(f"Annotation - cleaning command: {clean_command}")
5459                run_parallel_commands([clean_command], 1)

It takes a VCF file, annotates it with Annovar, and then updates the database with the new annotations

Parameters

threads: number of threads to use

Returns

the value of the variable "return_value".

def annotation_parquet(self, threads: int = None) -> None: View Source

5462    def annotation_parquet(self, threads: int = None) -> None:
5463        """
5464        It takes a VCF file, and annotates it with a parquet file
5465
5466        :param threads: number of threads to use for the annotation
5467        :return: the value of the variable "result".
5468        """
5469
5470        # DEBUG
5471        log.debug("Start annotation with parquet databases")
5472
5473        # Threads
5474        if not threads:
5475            threads = self.get_threads()
5476        log.debug("Threads: " + str(threads))
5477
5478        # DEBUG
5479        delete_tmp = True
5480        if self.get_config().get("verbosity", "warning") in ["debug"]:
5481            delete_tmp = False
5482            log.debug("Delete tmp files/folders: " + str(delete_tmp))
5483
5484        # Config
5485        databases_folders = set(
5486            self.get_config()
5487            .get("folders", {})
5488            .get("databases", {})
5489            .get("annotations", ["."])
5490            + self.get_config()
5491            .get("folders", {})
5492            .get("databases", {})
5493            .get("parquet", ["."])
5494        )
5495        log.debug("Databases annotations: " + str(databases_folders))
5496
5497        # Param
5498        annotations = (
5499            self.get_param()
5500            .get("annotation", {})
5501            .get("parquet", {})
5502            .get("annotations", None)
5503        )
5504        log.debug("Annotations: " + str(annotations))
5505
5506        # Assembly
5507        assembly = self.get_param().get(
5508            "assembly", self.get_config().get("assembly", DEFAULT_ASSEMBLY)
5509        )
5510
5511        # Force Update Annotation
5512        force_update_annotation = (
5513            self.get_param()
5514            .get("annotation", {})
5515            .get("options", {})
5516            .get("annotations_update", False)
5517        )
5518        log.debug(f"force_update_annotation={force_update_annotation}")
5519        force_append_annotation = (
5520            self.get_param()
5521            .get("annotation", {})
5522            .get("options", {})
5523            .get("annotations_append", False)
5524        )
5525        log.debug(f"force_append_annotation={force_append_annotation}")
5526
5527        # Data
5528        table_variants = self.get_table_variants()
5529
5530        # Check if not empty
5531        log.debug("Check if not empty")
5532        sql_query_chromosomes_df = self.get_query_to_df(
5533            f"""SELECT count(*) as count FROM {table_variants} as table_variants LIMIT 1"""
5534        )
5535        if not sql_query_chromosomes_df["count"][0]:
5536            log.info(f"VCF empty")
5537            return
5538
5539        # VCF header
5540        vcf_reader = self.get_header()
5541        log.debug("Initial header: " + str(vcf_reader.infos))
5542
5543        # Nb Variants POS
5544        log.debug("NB Variants Start")
5545        nb_variants = self.conn.execute(
5546            f"SELECT count(*) AS count FROM variants"
5547        ).fetchdf()["count"][0]
5548        log.debug("NB Variants Stop")
5549
5550        # Existing annotations
5551        for vcf_annotation in self.get_header().infos:
5552
5553            vcf_annotation_line = self.get_header().infos.get(vcf_annotation)
5554            log.debug(
5555                f"Existing annotations in VCF: {vcf_annotation} [{vcf_annotation_line}]"
5556            )
5557
5558        # Added columns
5559        added_columns = []
5560
5561        # drop indexes
5562        log.debug(f"Drop indexes...")
5563        self.drop_indexes()
5564
5565        if annotations:
5566
5567            if "ALL" in annotations:
5568
5569                all_param = annotations.get("ALL", {})
5570                all_param_formats = all_param.get("formats", None)
5571                all_param_releases = all_param.get("releases", None)
5572
5573                databases_infos_dict = self.scan_databases(
5574                    database_formats=all_param_formats,
5575                    database_releases=all_param_releases,
5576                )
5577                for database_infos in databases_infos_dict.keys():
5578                    if database_infos not in annotations:
5579                        annotations[database_infos] = {"INFO": None}
5580
5581            for annotation in annotations:
5582
5583                if annotation in ["ALL"]:
5584                    continue
5585
5586                # Annotation Name
5587                annotation_name = os.path.basename(annotation)
5588
5589                # Annotation fields
5590                annotation_fields = annotations[annotation]
5591                if not annotation_fields:
5592                    annotation_fields = {"INFO": None}
5593
5594                log.debug(f"Annotation '{annotation_name}'")
5595                log.debug(
5596                    f"Annotation '{annotation_name}' - fields: {annotation_fields}"
5597                )
5598
5599                # Create Database
5600                database = Database(
5601                    database=annotation,
5602                    databases_folders=databases_folders,
5603                    assembly=assembly,
5604                )
5605
5606                # Find files
5607                parquet_file = database.get_database()
5608                parquet_hdr_file = database.get_header_file()
5609                parquet_type = database.get_type()
5610
5611                # Check if files exists
5612                if not parquet_file or not parquet_hdr_file:
5613                    log.error("Annotation failed: file not found")
5614                    raise ValueError("Annotation failed: file not found")
5615                else:
5616                    # Get parquet connexion
5617                    parquet_sql_attach = database.get_sql_database_attach(
5618                        output="query"
5619                    )
5620                    if parquet_sql_attach:
5621                        self.conn.execute(parquet_sql_attach)
5622                    parquet_file_link = database.get_sql_database_link()
5623                    # Log
5624                    log.debug(
5625                        f"Annotation '{annotation_name}' - file: "
5626                        + str(parquet_file)
5627                        + " and "
5628                        + str(parquet_hdr_file)
5629                    )
5630
5631                    # Database full header columns
5632                    parquet_hdr_vcf_header_columns = database.get_header_file_columns(
5633                        parquet_hdr_file
5634                    )
5635                    # Log
5636                    log.debug(
5637                        "Annotation database header columns : "
5638                        + str(parquet_hdr_vcf_header_columns)
5639                    )
5640
5641                    # Load header as VCF object
5642                    parquet_hdr_vcf_header_infos = database.get_header().infos
5643                    # Log
5644                    log.debug(
5645                        "Annotation database header: "
5646                        + str(parquet_hdr_vcf_header_infos)
5647                    )
5648
5649                    # Get extra infos
5650                    parquet_columns = database.get_extra_columns()
5651                    # Log
5652                    log.debug("Annotation database Columns: " + str(parquet_columns))
5653
5654                    # Add extra columns if "ALL" in annotation_fields
5655                    # if "ALL" in annotation_fields:
5656                    #     allow_add_extra_column = True
5657                    if "ALL" in annotation_fields and database.get_extra_columns():
5658                        for extra_column in database.get_extra_columns():
5659                            if (
5660                                extra_column not in annotation_fields
5661                                and extra_column.replace("INFO/", "")
5662                                not in parquet_hdr_vcf_header_infos
5663                            ):
5664                                parquet_hdr_vcf_header_infos[extra_column] = (
5665                                    vcf.parser._Info(
5666                                        extra_column,
5667                                        ".",
5668                                        "String",
5669                                        f"{extra_column} description",
5670                                        "unknown",
5671                                        "unknown",
5672                                        self.code_type_map["String"],
5673                                    )
5674                                )
5675
5676                    # For all fields in database
5677                    annotation_fields_all = False
5678                    if "ALL" in annotation_fields or "INFO" in annotation_fields:
5679                        annotation_fields_all = True
5680                        annotation_fields = {
5681                            key: key for key in parquet_hdr_vcf_header_infos
5682                        }
5683
5684                        log.debug(
5685                            "Annotation database header - All annotations added: "
5686                            + str(annotation_fields)
5687                        )
5688
5689                    # Init
5690
5691                    # List of annotation fields to use
5692                    sql_query_annotation_update_info_sets = []
5693
5694                    # List of annotation to agregate
5695                    sql_query_annotation_to_agregate = []
5696
5697                    # Number of fields
5698                    nb_annotation_field = 0
5699
5700                    # Annotation fields processed
5701                    annotation_fields_processed = []
5702
5703                    # Columns mapping
5704                    map_columns = database.map_columns(
5705                        columns=annotation_fields, prefixes=["INFO/"]
5706                    )
5707
5708                    # Query dict for fields to remove (update option)
5709                    query_dict_remove = {}
5710
5711                    # Fetch Anotation fields
5712                    for annotation_field in annotation_fields:
5713
5714                        # annotation_field_column
5715                        annotation_field_column = map_columns.get(
5716                            annotation_field, "INFO"
5717                        )
5718
5719                        # field new name, if parametered
5720                        annotation_fields_new_name = annotation_fields.get(
5721                            annotation_field, annotation_field
5722                        )
5723                        if not annotation_fields_new_name:
5724                            annotation_fields_new_name = annotation_field
5725
5726                        # To annotate
5727                        # force_update_annotation = True
5728                        # force_append_annotation = True
5729                        # if annotation_field in parquet_hdr_vcf_header_infos and (force_update_annotation or (annotation_fields_new_name not in self.get_header().infos)):
5730                        if annotation_field in parquet_hdr_vcf_header_infos and (
5731                            force_update_annotation
5732                            or force_append_annotation
5733                            or (
5734                                annotation_fields_new_name
5735                                not in self.get_header().infos
5736                            )
5737                        ):
5738
5739                            # Add field to annotation to process list
5740                            annotation_fields_processed.append(
5741                                annotation_fields_new_name
5742                            )
5743
5744                            # explode infos for the field
5745                            annotation_fields_new_name_info_msg = ""
5746                            if (
5747                                force_update_annotation
5748                                and annotation_fields_new_name
5749                                in self.get_header().infos
5750                            ):
5751                                # Remove field from INFO
5752                                query = f"""
5753                                    UPDATE {table_variants} as table_variants
5754                                    SET INFO = REGEXP_REPLACE(
5755                                                concat(table_variants.INFO,''),
5756                                                ';*{annotation_fields_new_name}=[^;]*',
5757                                                ''
5758                                                )
5759                                    WHERE concat(';',table_variants.INFO) LIKE '%;{annotation_fields_new_name}=%'
5760                                """
5761                                annotation_fields_new_name_info_msg = " [update]"
5762                                query_dict_remove[
5763                                    f"remove 'INFO/{annotation_fields_new_name}'"
5764                                ] = query
5765
5766                            # Sep between fields in INFO
5767                            nb_annotation_field += 1
5768                            if nb_annotation_field > 1:
5769                                annotation_field_sep = ";"
5770                            else:
5771                                annotation_field_sep = ""
5772
5773                            log.info(
5774                                f"Annotation '{annotation_name}' - '{annotation_field}' -> '{annotation_fields_new_name}'{annotation_fields_new_name_info_msg}"
5775                            )
5776
5777                            # Add INFO field to header
5778                            parquet_hdr_vcf_header_infos_number = (
5779                                parquet_hdr_vcf_header_infos[annotation_field].num
5780                                or "."
5781                            )
5782                            parquet_hdr_vcf_header_infos_type = (
5783                                parquet_hdr_vcf_header_infos[annotation_field].type
5784                                or "String"
5785                            )
5786                            parquet_hdr_vcf_header_infos_description = (
5787                                parquet_hdr_vcf_header_infos[annotation_field].desc
5788                                or f"{annotation_field} description"
5789                            )
5790                            parquet_hdr_vcf_header_infos_source = (
5791                                parquet_hdr_vcf_header_infos[annotation_field].source
5792                                or "unknown"
5793                            )
5794                            parquet_hdr_vcf_header_infos_version = (
5795                                parquet_hdr_vcf_header_infos[annotation_field].version
5796                                or "unknown"
5797                            )
5798
5799                            vcf_reader.infos[annotation_fields_new_name] = (
5800                                vcf.parser._Info(
5801                                    annotation_fields_new_name,
5802                                    parquet_hdr_vcf_header_infos_number,
5803                                    parquet_hdr_vcf_header_infos_type,
5804                                    parquet_hdr_vcf_header_infos_description,
5805                                    parquet_hdr_vcf_header_infos_source,
5806                                    parquet_hdr_vcf_header_infos_version,
5807                                    self.code_type_map[
5808                                        parquet_hdr_vcf_header_infos_type
5809                                    ],
5810                                )
5811                            )
5812
5813                            # Append
5814                            if force_append_annotation:
5815                                query_case_when_append = f""" AND REGEXP_EXTRACT(concat(';', table_variants.INFO), ';{annotation_fields_new_name}=([^;]*)',1) IN ('','.') """
5816                            else:
5817                                query_case_when_append = ""
5818
5819                            # Annotation/Update query fields
5820                            # Found in INFO column
5821                            if (
5822                                annotation_field_column == "INFO"
5823                                and "INFO" in parquet_hdr_vcf_header_columns
5824                            ):
5825                                sql_query_annotation_update_info_sets.append(
5826                                    f"""
5827                                CASE WHEN REGEXP_EXTRACT(concat(';', table_parquet.INFO), ';{annotation_field}=([^;]*)',1) NOT IN ('','.') {query_case_when_append}
5828                                        THEN concat('{annotation_field_sep}', '{annotation_fields_new_name}=', REGEXP_EXTRACT(concat(';', table_parquet.INFO), ';{annotation_field}=([^;]*)',1))
5829                                        ELSE ''
5830                                    END
5831                                """
5832                                )
5833                            # Found in a specific column
5834                            else:
5835                                sql_query_annotation_update_info_sets.append(
5836                                    f"""
5837                                CASE WHEN CAST(table_parquet."{annotation_field_column}" AS VARCHAR) NOT IN ('','.') {query_case_when_append}
5838                                        THEN concat('{annotation_field_sep}', '{annotation_fields_new_name}=', replace(CAST(table_parquet."{annotation_field_column}" AS VARCHAR), ';', ','))
5839                                        ELSE ''
5840                                    END
5841                                """
5842                                )
5843                                sql_query_annotation_to_agregate.append(
5844                                    f""" string_agg(DISTINCT table_parquet_from."{annotation_field_column}", ',') AS "{annotation_field_column}" """
5845                                )
5846
5847                        # Not to annotate
5848                        else:
5849
5850                            if force_update_annotation:
5851                                annotation_message = "forced"
5852                            else:
5853                                annotation_message = "skipped"
5854
5855                            if annotation_field not in parquet_hdr_vcf_header_infos:
5856                                log.warning(
5857                                    f"Annotation '{annotation_name}' - '{annotation_field}' [{nb_annotation_field}] - not available in parquet file"
5858                                )
5859                            if annotation_fields_new_name in self.get_header().infos:
5860                                log.warning(
5861                                    f"Annotation '{annotation_name}' - '{annotation_fields_new_name}' [{nb_annotation_field}] - already exists in header ({annotation_message})"
5862                                )
5863
5864                    # Check if ALL fields have to be annotated. Thus concat all INFO field
5865                    # allow_annotation_full_info = True
5866                    allow_annotation_full_info = not force_append_annotation
5867
5868                    if parquet_type in ["regions"]:
5869                        allow_annotation_full_info = False
5870
5871                    if (
5872                        allow_annotation_full_info
5873                        and nb_annotation_field == len(annotation_fields)
5874                        and annotation_fields_all
5875                        and (
5876                            "INFO" in parquet_hdr_vcf_header_columns
5877                            and "INFO" in database.get_extra_columns()
5878                        )
5879                    ):
5880                        log.debug("Column INFO annotation enabled")
5881                        sql_query_annotation_update_info_sets = []
5882                        sql_query_annotation_update_info_sets.append(
5883                            f" table_parquet.INFO "
5884                        )
5885
5886                    if sql_query_annotation_update_info_sets:
5887
5888                        # Annotate
5889                        log.info(f"Annotation '{annotation_name}' - Annotation...")
5890
5891                        # Join query annotation update info sets for SQL
5892                        sql_query_annotation_update_info_sets_sql = ",".join(
5893                            sql_query_annotation_update_info_sets
5894                        )
5895
5896                        # Check chromosomes list (and variants infos)
5897                        sql_query_chromosomes = f"""
5898                            SELECT table_variants."#CHROM" as CHROM, count(*) AS count_variants, min(POS) AS min_variants, MAX(POS) AS max_variants
5899                            FROM {table_variants} as table_variants
5900                            GROUP BY table_variants."#CHROM"
5901                            ORDER BY table_variants."#CHROM"
5902                            """
5903                        sql_query_chromosomes_df = self.conn.execute(
5904                            sql_query_chromosomes
5905                        ).df()
5906                        sql_query_chromosomes_dict = {
5907                            entry["CHROM"]: {
5908                                "count": entry["count_variants"],
5909                                "min": entry["min_variants"],
5910                                "max": entry["max_variants"],
5911                            }
5912                            for index, entry in sql_query_chromosomes_df.iterrows()
5913                        }
5914
5915                        # Init
5916                        nb_of_query = 0
5917                        nb_of_variant_annotated = 0
5918                        query_dict = query_dict_remove
5919
5920                        # for chrom in sql_query_chromosomes_df["CHROM"]:
5921                        for chrom in sql_query_chromosomes_dict:
5922
5923                            # Number of variant by chromosome
5924                            nb_of_variant_by_chrom = sql_query_chromosomes_dict.get(
5925                                chrom, {}
5926                            ).get("count", 0)
5927
5928                            log.debug(
5929                                f"Annotation '{annotation_name}' - Chromosome '{chrom}' [{nb_of_variant_by_chrom} variants]..."
5930                            )
5931
5932                            # Annotation with regions database
5933                            if parquet_type in ["regions"]:
5934                                sql_query_annotation_from_clause = f"""
5935                                    FROM (
5936                                        SELECT 
5937                                            '{chrom}' AS \"#CHROM\",
5938                                            table_variants_from.\"POS\" AS \"POS\",
5939                                            {",".join(sql_query_annotation_to_agregate)}
5940                                        FROM {table_variants} as table_variants_from
5941                                        LEFT JOIN {parquet_file_link} as table_parquet_from ON (
5942                                            table_parquet_from."#CHROM" = '{chrom}'
5943                                            AND table_variants_from.\"POS\" <= table_parquet_from.\"END\"
5944                                            AND (table_variants_from.\"POS\" >= (table_parquet_from.\"START\"+1)
5945                                                OR table_variants_from.\"POS\" + (len(table_variants_from.\"REF\")-1) >= (table_parquet_from.\"START\"+1)
5946                                                )
5947                                        )
5948                                        WHERE table_variants_from.\"#CHROM\" in ('{chrom}')
5949                                        GROUP BY table_variants_from.\"POS\"
5950                                        )
5951                                        as table_parquet
5952                                """
5953
5954                                sql_query_annotation_where_clause = """
5955                                    table_parquet.\"#CHROM\" = table_variants.\"#CHROM\"
5956                                    AND table_parquet.\"POS\" = table_variants.\"POS\"
5957                                """
5958
5959                            # Annotation with variants database
5960                            else:
5961                                sql_query_annotation_from_clause = f"""
5962                                    FROM {parquet_file_link} as table_parquet
5963                                """
5964                                sql_query_annotation_where_clause = f"""
5965                                    table_variants."#CHROM" = '{chrom}'
5966                                    AND table_parquet.\"#CHROM\" = table_variants.\"#CHROM\" 
5967                                    AND table_parquet.\"POS\" = table_variants.\"POS\"
5968                                    AND table_parquet.\"ALT\" = table_variants.\"ALT\"
5969                                    AND table_parquet.\"REF\" = table_variants.\"REF\"
5970                                """
5971
5972                            # Create update query
5973                            sql_query_annotation_chrom_interval_pos = f"""
5974                                UPDATE {table_variants} as table_variants
5975                                    SET INFO = 
5976                                        concat(
5977                                            CASE WHEN table_variants.INFO NOT IN ('','.')
5978                                                THEN table_variants.INFO
5979                                                ELSE ''
5980                                            END
5981                                            ,
5982                                            CASE WHEN table_variants.INFO NOT IN ('','.')
5983                                                        AND (
5984                                                        concat({sql_query_annotation_update_info_sets_sql})
5985                                                        )
5986                                                        NOT IN ('','.') 
5987                                                    THEN ';'
5988                                                    ELSE ''
5989                                            END
5990                                            ,
5991                                            {sql_query_annotation_update_info_sets_sql}
5992                                            )
5993                                    {sql_query_annotation_from_clause}
5994                                    WHERE {sql_query_annotation_where_clause}
5995                                    ;
5996                                """
5997
5998                            # Add update query to dict
5999                            query_dict[
6000                                f"{chrom} [{nb_of_variant_by_chrom} variants]"
6001                            ] = sql_query_annotation_chrom_interval_pos
6002
6003                        nb_of_query = len(query_dict)
6004                        num_query = 0
6005
6006                        # SET max_expression_depth TO x
6007                        self.conn.execute("SET max_expression_depth TO 10000")
6008
6009                        for query_name in query_dict:
6010                            query = query_dict[query_name]
6011                            num_query += 1
6012                            log.info(
6013                                f"Annotation '{annotation_name}' - Annotation - Query [{num_query}/{nb_of_query}] {query_name}..."
6014                            )
6015                            result = self.conn.execute(query)
6016                            nb_of_variant_annotated_by_query = result.df()["Count"][0]
6017                            nb_of_variant_annotated += nb_of_variant_annotated_by_query
6018                            log.info(
6019                                f"Annotation '{annotation_name}' - Annotation - Query [{num_query}/{nb_of_query}] {query_name} - {nb_of_variant_annotated_by_query} variants annotated"
6020                            )
6021
6022                        log.info(
6023                            f"Annotation '{annotation_name}' - Annotation of {nb_of_variant_annotated} variants out of {nb_variants} (with {nb_of_query} queries)"
6024                        )
6025
6026                    else:
6027
6028                        log.info(
6029                            f"Annotation '{annotation_name}' - No Annotations available"
6030                        )
6031
6032                    log.debug("Final header: " + str(vcf_reader.infos))
6033
6034        # Remove added columns
6035        for added_column in added_columns:
6036            self.drop_column(column=added_column)

It takes a VCF file, and annotates it with a parquet file

Parameters

threads: number of threads to use for the annotation

Returns

the value of the variable "result".

def annotation_splice(self, threads: int = None) -> None: View Source

6038    def annotation_splice(self, threads: int = None) -> None:
6039        """
6040        This function annotate with snpEff
6041
6042        :param threads: The number of threads to use
6043        :return: the value of the variable "return_value".
6044        """
6045
6046        # DEBUG
6047        log.debug("Start annotation with splice tools")
6048
6049        # Threads
6050        if not threads:
6051            threads = self.get_threads()
6052        log.debug("Threads: " + str(threads))
6053
6054        # DEBUG
6055        delete_tmp = True
6056        if self.get_config().get("verbosity", "warning") in ["debug"]:
6057            delete_tmp = False
6058            log.debug("Delete tmp files/folders: " + str(delete_tmp))
6059
6060        # Config
6061        config = self.get_config()
6062        log.debug("Config: " + str(config))
6063        splice_config = config.get("tools", {}).get("splice", {})
6064        if not splice_config:
6065            splice_config = DEFAULT_TOOLS_BIN.get("splice", {})
6066        if not splice_config:
6067            msg_err = "No Splice tool config"
6068            log.error(msg_err)
6069            raise ValueError(msg_err)
6070        log.debug(f"splice_config={splice_config}")
6071
6072        # Config - Folders - Databases
6073        databases_folders = (
6074            config.get("folders", {}).get("databases", {}).get("splice", ["."])
6075        )
6076        log.debug("Databases annotations: " + str(databases_folders))
6077
6078        # Splice docker image
6079        splice_docker_image = splice_config.get("docker").get("image")
6080
6081        # Pull splice image if it's not already there
6082        if not check_docker_image_exists(splice_docker_image):
6083            log.warning(
6084                f"Annotation: splice docker image {splice_docker_image} not found locally, trying to pull from dockerhub"
6085            )
6086            try:
6087                command(f"docker pull {splice_config.get('docker').get('image')}")
6088            except subprocess.CalledProcessError:
6089                msg_err = f"Unable to find docker {splice_docker_image} on dockerhub"
6090                log.error(msg_err)
6091                raise ValueError(msg_err)
6092                return None
6093
6094        # Config - splice databases
6095        splice_databases = (
6096            config.get("folders", {})
6097            .get("databases", {})
6098            .get("splice", DEFAULT_SPLICE_FOLDER)
6099        )
6100        splice_databases = full_path(splice_databases)
6101
6102        # Param
6103        param = self.get_param()
6104        log.debug("Param: " + str(param))
6105
6106        # Param
6107        options = param.get("annotation", {}).get("splice", {})
6108        log.debug("Options: " + str(options))
6109
6110        # Data
6111        table_variants = self.get_table_variants()
6112
6113        # Check if not empty
6114        log.debug("Check if not empty")
6115        sql_query_chromosomes = (
6116            f"""SELECT count(*) as count FROM {table_variants} as table_variants"""
6117        )
6118        if not self.get_query_to_df(f"{sql_query_chromosomes}")["count"][0]:
6119            log.info("VCF empty")
6120            return None
6121
6122        # Export in VCF
6123        log.debug("Create initial file to annotate")
6124
6125        # Create output folder
6126        output_folder = os.path.join(self.get_tmp_dir(), f"splice-{get_random()}")
6127        if not os.path.exists(output_folder):
6128            Path(output_folder).mkdir(parents=True, exist_ok=True)
6129
6130        # Create tmp VCF file
6131        tmp_vcf = NamedTemporaryFile(
6132            prefix=self.get_prefix(),
6133            dir=output_folder,
6134            suffix=".vcf",
6135            delete=False,
6136        )
6137        tmp_vcf_name = tmp_vcf.name
6138
6139        # VCF header
6140        header = self.get_header()
6141
6142        # Existing annotations
6143        for vcf_annotation in self.get_header().infos:
6144
6145            vcf_annotation_line = self.get_header().infos.get(vcf_annotation)
6146            log.debug(
6147                f"Existing annotations in VCF: {vcf_annotation} [{vcf_annotation_line}]"
6148            )
6149
6150        # Memory limit
6151        if config.get("memory", None):
6152            memory_limit = config.get("memory", "8G").upper()
6153            # upper()
6154        else:
6155            memory_limit = "8G"
6156        log.debug(f"memory_limit: {memory_limit}")
6157
6158        # Check number of variants to annotate
6159        where_clause_regex_spliceai = r"SpliceAI_\w+"
6160        where_clause_regex_spip = r"SPiP_\w+"
6161        where_clause = f""" WHERE NOT regexp_matches("INFO", '{where_clause_regex_spliceai}') AND NOT regexp_matches("INFO", '{where_clause_regex_spip}')"""
6162        df_list_of_variants_to_annotate = self.get_query_to_df(
6163            query=f""" SELECT * FROM variants {where_clause} """
6164        )
6165        if len(df_list_of_variants_to_annotate) == 0:
6166            log.warning(
6167                f"No variants to annotate with splice. Variants probably already annotated with splice"
6168            )
6169            return None
6170        else:
6171            log.info(f"Annotation: {len(df_list_of_variants_to_annotate)} variants")
6172
6173        # Export VCF file
6174        self.export_variant_vcf(
6175            vcf_file=tmp_vcf_name,
6176            remove_info=True,
6177            add_samples=True,
6178            index=False,
6179            where_clause=where_clause,
6180        )
6181
6182        # Create docker container and launch splice analysis
6183        if splice_config:
6184
6185            # Splice mount folders
6186            mount_folders = splice_config.get("mount", {})
6187
6188            # Genome mount
6189            mount_folders[
6190                config.get("folders", {})
6191                .get("databases", {})
6192                .get("genomes", DEFAULT_GENOME_FOLDER)
6193            ] = "ro"
6194
6195            # SpliceAI mount
6196            mount_folders[
6197                config.get("folders", {})
6198                .get("databases", {})
6199                .get("spliceai", DEFAULT_SPLICEAI_FOLDER)
6200            ] = "ro"
6201
6202            # Genome mount
6203            mount_folders[
6204                config.get("folders", {})
6205                .get("databases", {})
6206                .get("spip", DEFAULT_SPIP_FOLDER)
6207            ] = "ro"
6208
6209            # Mount folders
6210            mount = []
6211
6212            # Config mount
6213            mount = [
6214                f"-v {full_path(path)}:{full_path(path)}:{mode}"
6215                for path, mode in mount_folders.items()
6216            ]
6217
6218            if any(value for value in splice_config.values() if value is None):
6219                log.warning("At least one splice config parameter is empty")
6220                return None
6221
6222            # Params in splice nf
6223            def check_values(dico: dict):
6224                """
6225                Ensure parameters for NF splice pipeline
6226                """
6227                for key, val in dico.items():
6228                    if key == "genome":
6229                        if any(
6230                            assemb in options.get("genome", {})
6231                            for assemb in ["hg19", "GRCh37", "grch37", "GRCH37"]
6232                        ):
6233                            yield f"--{key} hg19"
6234                        elif any(
6235                            assemb in options.get("genome", {})
6236                            for assemb in ["hg38", "GRCh38", "grch38", "GRCH38"]
6237                        ):
6238                            yield f"--{key} hg38"
6239                    elif (
6240                        (isinstance(val, str) and val)
6241                        or isinstance(val, int)
6242                        or isinstance(val, bool)
6243                    ):
6244                        yield f"--{key} {val}"
6245
6246            # Genome
6247            genome = options.get("genome", config.get("assembly", DEFAULT_ASSEMBLY))
6248            options["genome"] = genome
6249
6250            # NF params
6251            nf_params = []
6252
6253            # Add options
6254            if options:
6255                nf_params = list(check_values(options))
6256                log.debug(f"Splice NF params: {' '.join(nf_params)}")
6257            else:
6258                log.debug("No NF params provided")
6259
6260            # Add threads
6261            if "threads" not in options.keys():
6262                nf_params.append(f"--threads {threads}")
6263
6264            # Genome path
6265            genome_path = find_genome(
6266                config.get("folders", {})
6267                .get("databases", {})
6268                .get("genomes", DEFAULT_GENOME_FOLDER),
6269                file=f"{genome}.fa",
6270            )
6271            # Add genome path
6272            if not genome_path:
6273                raise ValueError(
6274                    f"Can't find genome assembly {genome}.fa in {config.get('folders', {}).get('databases', {}).get('genomes', DEFAULT_GENOME_FOLDER)}"
6275                )
6276            else:
6277                log.debug(f"Genome: {genome_path}")
6278                nf_params.append(f"--genome_path {genome_path}")
6279
6280            def splice_annotations(options: dict = {}, config: dict = {}) -> list:
6281                """
6282                Setting up updated databases for SPiP and SpliceAI
6283                """
6284
6285                try:
6286
6287                    # SpliceAI assembly transcriptome
6288                    spliceai_assembly = os.path.join(
6289                        config.get("folders", {})
6290                        .get("databases", {})
6291                        .get("spliceai", {}),
6292                        options.get("genome"),
6293                        "transcriptome",
6294                    )
6295                    spip_assembly = options.get("genome")
6296
6297                    spip = find(
6298                        f"transcriptome_{spip_assembly}.RData",
6299                        config.get("folders", {}).get("databases", {}).get("spip", {}),
6300                    )
6301                    spliceai = find("spliceai.refseq.txt", spliceai_assembly)
6302                    log.debug(f"SPiP annotations: {spip}")
6303                    log.debug(f"SpliceAI annotations: {spliceai}")
6304                    if spip and spliceai:
6305                        return [
6306                            f"--spip_transcriptome {spip}",
6307                            f"--spliceai_annotations {spliceai}",
6308                        ]
6309                    else:
6310                        # TODO crash and go on with basic annotations ?
6311                        # raise ValueError(
6312                        #     "Can't find splice databases in configuration EXIT"
6313                        # )
6314                        log.warning(
6315                            "Can't find splice databases in configuration, use annotations file from image"
6316                        )
6317                except TypeError:
6318                    log.warning(
6319                        "Can't find splice databases in configuration, use annotations file from image"
6320                    )
6321                    return []
6322
6323            # Add options, check if transcriptome option have already beend provided
6324            if (
6325                "spip_transcriptome" not in nf_params
6326                and "spliceai_transcriptome" not in nf_params
6327            ):
6328                splice_reference = splice_annotations(options, config)
6329                if splice_reference:
6330                    nf_params.extend(splice_reference)
6331
6332            nf_params.append(f"--output_folder {output_folder}")
6333
6334            random_uuid = f"HOWARD-SPLICE-{get_random()}"
6335            cmd = f"nextflow -log {os.path.join(output_folder, f'{random_uuid}.log')} -c /app/SpliceToolBox/src/splicetoolbox/nextflow/nextflow.docker.config run /app/SpliceToolBox/src/splicetoolbox/nextflow/main.nf -entry SPLICE --vcf {tmp_vcf_name} {' '.join(nf_params)} -profile standard,conda,singularity,report,timeline"
6336            log.debug(cmd)
6337
6338            splice_config["docker"]["command"] = cmd
6339
6340            docker_cmd = get_bin_command(
6341                tool="splice",
6342                bin_type="docker",
6343                config=config,
6344                default_folder=f"{DEFAULT_TOOLS_FOLDER}/docker",
6345                add_options=f"--name {random_uuid} {' '.join(mount)}",
6346            )
6347
6348            # Docker debug
6349            # if splice_config.get("rm_container"):
6350            #     rm_container = "--rm"
6351            # else:
6352            #     rm_container = ""
6353            # docker_cmd = f"docker run {rm_container} --entrypoint '/bin/bash' --name {random_uuid} {' '.join(mount)} {':'.join(splice_config.get('image'))} {cmd}"
6354
6355            log.debug(docker_cmd)
6356            res = subprocess.run(docker_cmd, shell=True, capture_output=True, text=True)
6357            log.debug(res.stdout)
6358            if res.stderr:
6359                log.error(res.stderr)
6360            res.check_returncode()
6361        else:
6362            log.warning(f"Splice tool configuration not found: {config}")
6363
6364        # Update variants
6365        log.info("Annotation - Updating...")
6366        # Test find output vcf
6367        log.debug(
6368            f"TMP splice output: {os.path.basename(tmp_vcf_name).replace('.vcf', '')}.spip.spliceai.sorted.vcf.gz"
6369        )
6370        output_vcf = []
6371        # Wrong folder to look in
6372        for files in os.listdir(os.path.dirname(tmp_vcf_name)):
6373            if (
6374                files
6375                == f"{os.path.basename(tmp_vcf_name).replace('.vcf', '')}.spip.spliceai.sorted.vcf.gz"
6376            ):
6377                output_vcf.append(os.path.join(os.path.dirname(tmp_vcf_name), files))
6378        # log.debug(os.listdir(options.get("output_folder")))
6379        log.debug(f"Splice annotated vcf: {output_vcf[0]}")
6380        if not output_vcf:
6381            log.debug(
6382                f"Splice output was not generated {os.path.basename(tmp_vcf_name)}*.spip.spliceai.sorted.vcf.gz"
6383            )
6384        else:
6385            # Get new header from annotated vcf
6386            log.debug(f"Initial header: {len(header.infos)} fields")
6387            # Create new header with splice infos
6388            new_vcf = Variants(input=output_vcf[0])
6389            new_vcf_header = new_vcf.get_header().infos
6390            for keys, infos in new_vcf_header.items():
6391                if keys not in header.infos.keys():
6392                    header.infos[keys] = infos
6393            log.debug(f"New header: {len(header.infos)} fields")
6394            log.debug(f"Splice tmp output: {output_vcf[0]}")
6395            self.update_from_vcf(output_vcf[0])
6396
6397        # Remove folder
6398        remove_if_exists(output_folder)

This function annotate with snpEff

Parameters

threads: The number of threads to use

Returns

the value of the variable "return_value".

def get_config_default(self, name: str) -> dict: View Source

6404    def get_config_default(self, name: str) -> dict:
6405        """
6406        The function `get_config_default` returns a dictionary containing default configurations for
6407        various calculations and prioritizations.
6408
6409        :param name: The `get_config_default` function returns a dictionary containing default
6410        configurations for different calculations and prioritizations. The `name` parameter is used to
6411        specify which specific configuration to retrieve from the dictionary
6412        :type name: str
6413        :return: The function `get_config_default` returns a dictionary containing default configuration
6414        settings for different calculations and prioritizations. The specific configuration settings are
6415        retrieved based on the input `name` parameter provided to the function. If the `name` parameter
6416        matches a key in the `config_default` dictionary, the corresponding configuration settings are
6417        returned. If there is no match, an empty dictionary is returned.
6418        """
6419
6420        config_default = {
6421            "calculations": {
6422                "variant_chr_pos_alt_ref": {
6423                    "type": "sql",
6424                    "name": "variant_chr_pos_alt_ref",
6425                    "description": "Create a variant ID with chromosome, position, alt and ref",
6426                    "available": False,
6427                    "output_column_name": "variant_chr_pos_alt_ref",
6428                    "output_column_type": "String",
6429                    "output_column_description": "variant ID with chromosome, position, alt and ref",
6430                    "operation_query": """ concat("#CHROM", '_', "POS", '_', "REF", '_', "ALT") """,
6431                    "operation_info": True,
6432                },
6433                "VARTYPE": {
6434                    "type": "sql",
6435                    "name": "VARTYPE",
6436                    "description": "Variant type (e.g. SNV, INDEL, MNV, BND...)",
6437                    "available": True,
6438                    "output_column_name": "VARTYPE",
6439                    "output_column_type": "String",
6440                    "output_column_description": "Variant type: SNV if X>Y, MOSAIC if X>Y,Z or X,Y>Z, INDEL if XY>Z or X>YZ",
6441                    "operation_query": """
6442                            CASE
6443                                WHEN "SVTYPE" NOT NULL THEN "SVTYPE"
6444                                WHEN LENGTH(REF) = 1 AND LENGTH(ALT) = 1 THEN 'SNV'
6445                                WHEN REF LIKE '%,%' OR ALT LIKE '%,%' THEN 'MOSAIC'
6446                                WHEN LENGTH(REF) == LENGTH(ALT) AND LENGTH(REF) > 1 THEN 'MNV'
6447                                WHEN LENGTH(REF) <> LENGTH(ALT) THEN 'INDEL'
6448                                ELSE 'UNDEFINED'
6449                            END
6450                            """,
6451                    "info_fields": ["SVTYPE"],
6452                    "operation_info": True,
6453                },
6454                "snpeff_hgvs": {
6455                    "type": "python",
6456                    "name": "snpeff_hgvs",
6457                    "description": "HGVS nomenclatures from snpEff annotation",
6458                    "available": True,
6459                    "function_name": "calculation_extract_snpeff_hgvs",
6460                    "function_params": ["snpeff_hgvs", "ANN"],
6461                },
6462                "snpeff_ann_explode": {
6463                    "type": "python",
6464                    "name": "snpeff_ann_explode",
6465                    "description": "Explode snpEff annotations with uniquify values",
6466                    "available": True,
6467                    "function_name": "calculation_snpeff_ann_explode",
6468                    "function_params": [False, "fields", "snpeff_", "ANN"],
6469                },
6470                "snpeff_ann_explode_uniquify": {
6471                    "type": "python",
6472                    "name": "snpeff_ann_explode_uniquify",
6473                    "description": "Explode snpEff annotations",
6474                    "available": True,
6475                    "function_name": "calculation_snpeff_ann_explode",
6476                    "function_params": [True, "fields", "snpeff_uniquify_", "ANN"],
6477                },
6478                "snpeff_ann_explode_json": {
6479                    "type": "python",
6480                    "name": "snpeff_ann_explode_json",
6481                    "description": "Explode snpEff annotations in JSON format",
6482                    "available": True,
6483                    "function_name": "calculation_snpeff_ann_explode",
6484                    "function_params": [False, "JSON", "snpeff_json", "ANN"],
6485                },
6486                "NOMEN": {
6487                    "type": "python",
6488                    "name": "NOMEN",
6489                    "description": "NOMEN information (e.g. NOMEN, CNOMEN, PNOMEN...) from HGVS nomenclature field",
6490                    "available": True,
6491                    "function_name": "calculation_extract_nomen",
6492                    "function_params": [],
6493                },
6494                "FINDBYPIPELINE": {
6495                    "type": "python",
6496                    "name": "FINDBYPIPELINE",
6497                    "description": "Number of pipeline that identify the variant (for multi pipeline VCF)",
6498                    "available": True,
6499                    "function_name": "calculation_find_by_pipeline",
6500                    "function_params": ["findbypipeline"],
6501                },
6502                "FINDBYSAMPLE": {
6503                    "type": "python",
6504                    "name": "FINDBYSAMPLE",
6505                    "description": "Number of sample that have a genotype for the variant (for multi sample VCF)",
6506                    "available": True,
6507                    "function_name": "calculation_find_by_pipeline",
6508                    "function_params": ["findbysample"],
6509                },
6510                "GENOTYPECONCORDANCE": {
6511                    "type": "python",
6512                    "name": "GENOTYPECONCORDANCE",
6513                    "description": "Concordance of genotype for multi caller VCF",
6514                    "available": True,
6515                    "function_name": "calculation_genotype_concordance",
6516                    "function_params": [],
6517                },
6518                "BARCODE": {
6519                    "type": "python",
6520                    "name": "BARCODE",
6521                    "description": "BARCODE as VaRank tool",
6522                    "available": True,
6523                    "function_name": "calculation_barcode",
6524                    "function_params": [],
6525                },
6526                "BARCODEFAMILY": {
6527                    "type": "python",
6528                    "name": "BARCODEFAMILY",
6529                    "description": "BARCODEFAMILY as VaRank tool",
6530                    "available": True,
6531                    "function_name": "calculation_barcode_family",
6532                    "function_params": ["BCF"],
6533                },
6534                "TRIO": {
6535                    "type": "python",
6536                    "name": "TRIO",
6537                    "description": "Inheritance for a trio family",
6538                    "available": True,
6539                    "function_name": "calculation_trio",
6540                    "function_params": [],
6541                },
6542                "VAF": {
6543                    "type": "python",
6544                    "name": "VAF",
6545                    "description": "Variant Allele Frequency (VAF) harmonization",
6546                    "available": True,
6547                    "function_name": "calculation_vaf_normalization",
6548                    "function_params": [],
6549                },
6550                "VAF_stats": {
6551                    "type": "python",
6552                    "name": "VAF_stats",
6553                    "description": "Variant Allele Frequency (VAF) statistics",
6554                    "available": True,
6555                    "function_name": "calculation_genotype_stats",
6556                    "function_params": ["VAF"],
6557                },
6558                "DP_stats": {
6559                    "type": "python",
6560                    "name": "DP_stats",
6561                    "description": "Depth (DP) statistics",
6562                    "available": True,
6563                    "function_name": "calculation_genotype_stats",
6564                    "function_params": ["DP"],
6565                },
6566                "variant_id": {
6567                    "type": "python",
6568                    "name": "variant_id",
6569                    "description": "Variant ID generated from variant position and type",
6570                    "available": True,
6571                    "function_name": "calculation_variant_id",
6572                    "function_params": [],
6573                },
6574                "transcripts_json": {
6575                    "type": "python",
6576                    "name": "transcripts_json",
6577                    "description": "Add transcripts annotations in JSON format (field 'transcripts_json')",
6578                    "available": True,
6579                    "function_name": "calculation_transcripts_annotation",
6580                    "function_params": ["transcripts_json", None],
6581                },
6582                "transcripts_ann": {
6583                    "type": "python",
6584                    "name": "transcripts_ann",
6585                    "description": "Add transcripts annotations in structured format (field 'transcripts_ann')",
6586                    "available": True,
6587                    "function_name": "calculation_transcripts_annotation",
6588                    "function_params": [None, "transcripts_ann"],
6589                },
6590                "transcripts_annotations": {
6591                    "type": "python",
6592                    "name": "transcripts_annotations",
6593                    "description": "Add transcripts annotations in JSON and/or structured format (see param JSON file)",
6594                    "available": True,
6595                    "function_name": "calculation_transcripts_annotation",
6596                    "function_params": [None, None],
6597                },
6598                "transcripts_prioritization": {
6599                    "type": "python",
6600                    "name": "transcripts_prioritization",
6601                    "description": "Prioritize transcripts with a prioritization profile (using param.json)",
6602                    "available": True,
6603                    "function_name": "calculation_transcripts_prioritization",
6604                    "function_params": [],
6605                },
6606            },
6607            "prioritizations": {
6608                "default": {
6609                    "ANN2": [
6610                        {
6611                            "type": "contains",
6612                            "value": "HIGH",
6613                            "score": 5,
6614                            "flag": "PASS",
6615                            "comment": [
6616                                "The variant is assumed to have high (disruptive) impact in the protein, probably causing protein truncation, loss of function or triggering nonsense mediated decay"
6617                            ],
6618                        },
6619                        {
6620                            "type": "contains",
6621                            "value": "MODERATE",
6622                            "score": 3,
6623                            "flag": "PASS",
6624                            "comment": [
6625                                "A non-disruptive variant that might change protein effectiveness"
6626                            ],
6627                        },
6628                        {
6629                            "type": "contains",
6630                            "value": "LOW",
6631                            "score": 0,
6632                            "flag": "FILTERED",
6633                            "comment": [
6634                                "Assumed to be mostly harmless or unlikely to change protein behavior"
6635                            ],
6636                        },
6637                        {
6638                            "type": "contains",
6639                            "value": "MODIFIER",
6640                            "score": 0,
6641                            "flag": "FILTERED",
6642                            "comment": [
6643                                "Usually non-coding variants or variants affecting non-coding genes, where predictions are difficult or there is no evidence of impact"
6644                            ],
6645                        },
6646                    ],
6647                }
6648            },
6649        }
6650
6651        return config_default.get(name, None)

The function get_config_default returns a dictionary containing default configurations for various calculations and prioritizations.

Parameters

name: The get_config_default function returns a dictionary containing default configurations for different calculations and prioritizations. The name parameter is used to specify which specific configuration to retrieve from the dictionary

Returns

The function get_config_default returns a dictionary containing default configuration settings for different calculations and prioritizations. The specific configuration settings are retrieved based on the input name parameter provided to the function. If the name parameter matches a key in the config_default dictionary, the corresponding configuration settings are returned. If there is no match, an empty dictionary is returned.

def get_config_json(self, name: str, config_dict: dict = {}, config_file: str = None) -> dict: View Source

6653    def get_config_json(
6654        self, name: str, config_dict: dict = {}, config_file: str = None
6655    ) -> dict:
6656        """
6657        The function `get_config_json` retrieves a configuration JSON object with prioritizations from
6658        default values, a dictionary, and a file.
6659
6660        :param name: The `name` parameter in the `get_config_json` function is a string that represents
6661        the name of the configuration. It is used to identify and retrieve the configuration settings
6662        for a specific component or module
6663        :type name: str
6664        :param config_dict: The `config_dict` parameter in the `get_config_json` function is a
6665        dictionary that allows you to provide additional configuration settings or overrides. When you
6666        call the `get_config_json` function, you can pass a dictionary containing key-value pairs where
6667        the key is the configuration setting you want to override or
6668        :type config_dict: dict
6669        :param config_file: The `config_file` parameter in the `get_config_json` function is used to
6670        specify the path to a configuration file that contains additional settings. If provided, the
6671        function will read the contents of this file and update the configuration dictionary with the
6672        values found in the file, overriding any existing values with the
6673        :type config_file: str
6674        :return: The function `get_config_json` returns a dictionary containing the configuration
6675        settings.
6676        """
6677
6678        # Create with default prioritizations
6679        config_default = self.get_config_default(name=name)
6680        configuration = config_default
6681        # log.debug(f"configuration={configuration}")
6682
6683        # Replace prioritizations from dict
6684        for config in config_dict:
6685            configuration[config] = config_dict[config]
6686
6687        # Replace prioritizations from file
6688        config_file = full_path(config_file)
6689        if config_file:
6690            if os.path.exists(config_file):
6691                with open(config_file) as config_file_content:
6692                    config_file_dict = json.load(config_file_content)
6693                for config in config_file_dict:
6694                    configuration[config] = config_file_dict[config]
6695            else:
6696                msg_error = f"Config '{name}' file '{config_file}' does NOT exist"
6697                log.error(msg_error)
6698                raise ValueError(msg_error)
6699
6700        return configuration

The function get_config_json retrieves a configuration JSON object with prioritizations from default values, a dictionary, and a file.

Parameters

name: The name parameter in the get_config_json function is a string that represents the name of the configuration. It is used to identify and retrieve the configuration settings for a specific component or module
config_dict: The config_dict parameter in the get_config_json function is a dictionary that allows you to provide additional configuration settings or overrides. When you call the get_config_json function, you can pass a dictionary containing key-value pairs where the key is the configuration setting you want to override or
config_file: The config_file parameter in the get_config_json function is used to specify the path to a configuration file that contains additional settings. If provided, the function will read the contents of this file and update the configuration dictionary with the values found in the file, overriding any existing values with the

Returns

The function get_config_json returns a dictionary containing the configuration settings.

def prioritization( self, table: str = None, pz_prefix: str = None, pz_param: dict = None) -> bool: View Source

6702    def prioritization(
6703        self, table: str = None, pz_prefix: str = None, pz_param: dict = None
6704    ) -> bool:
6705        """
6706        The `prioritization` function in Python processes VCF files, adds new INFO fields, and
6707        prioritizes variants based on configured profiles and criteria.
6708
6709        :param table: The `table` parameter in the `prioritization` function is used to specify the name
6710        of the table (presumably a VCF file) on which the prioritization operation will be performed. If
6711        a table name is provided, the method will prioritize the variants in that specific table
6712        :type table: str
6713        :param pz_prefix: The `pz_prefix` parameter is used to specify a prefix that will be added to
6714        certain INFO fields in a VCF file during the prioritization process. If this parameter is not
6715        provided, the code will use a default prefix value of "PZ"
6716        :type pz_prefix: str
6717        :param pz_param: The `pz_param` parameter in the `prioritization` method is used to pass
6718        additional parameters specific to the prioritization process. These parameters can include
6719        settings related to prioritization profiles, fields, scoring modes, flags, comments, and other
6720        configurations needed for the prioritization of variants in a V
6721        :type pz_param: dict
6722        :return: A boolean value (True) is being returned from the `prioritization` function.
6723        """
6724
6725        # Config
6726        config = self.get_config()
6727
6728        # Param
6729        param = self.get_param()
6730
6731        # Prioritization param
6732        if pz_param is not None:
6733            prioritization_param = pz_param
6734        else:
6735            prioritization_param = param.get("prioritization", {})
6736
6737        # Configuration profiles
6738        prioritization_config_file = prioritization_param.get(
6739            "prioritization_config", None
6740        )
6741        prioritization_config_file = full_path(prioritization_config_file)
6742        prioritizations_config = self.get_config_json(
6743            name="prioritizations", config_file=prioritization_config_file
6744        )
6745
6746        # Prioritization prefix
6747        pz_prefix_default = "PZ"
6748        if pz_prefix is None:
6749            pz_prefix = prioritization_param.get("pzprefix", pz_prefix_default)
6750
6751        # Prioritization options
6752        profiles = prioritization_param.get("profiles", [])
6753        if isinstance(profiles, str):
6754            profiles = profiles.split(",")
6755        pzfields = prioritization_param.get(
6756            "pzfields", [f"{pz_prefix}Flag", f"{pz_prefix}Score"]
6757        )
6758        if isinstance(pzfields, str):
6759            pzfields = pzfields.split(",")
6760        default_profile = prioritization_param.get("default_profile", None)
6761        pzfields_sep = prioritization_param.get("pzfields_sep", "_")
6762        prioritization_score_mode = prioritization_param.get(
6763            "prioritization_score_mode", "HOWARD"
6764        )
6765
6766        # Quick Prioritizations
6767        prioritizations = param.get("prioritizations", None)
6768        if prioritizations:
6769            log.info("Quick Prioritization:")
6770            for profile in prioritizations.split(","):
6771                if profile not in profiles:
6772                    profiles.append(profile)
6773                    log.info(f"   {profile}")
6774
6775        # If profile "ALL" provided, all profiles in the config profiles
6776        if "ALL" in profiles:
6777            profiles = list(prioritizations_config.keys())
6778
6779        for profile in profiles:
6780            if prioritizations_config.get(profile, None):
6781                log.debug(f"Profile '{profile}' configured")
6782            else:
6783                msg_error = f"Profile '{profile}' NOT configured"
6784                log.error(msg_error)
6785                raise ValueError(msg_error)
6786
6787        if profiles:
6788            log.info(f"Prioritization... ")
6789        else:
6790            log.debug(f"No profile defined")
6791            return False
6792
6793        if not default_profile and len(profiles):
6794            default_profile = profiles[0]
6795
6796        log.debug("Profiles availables: " + str(list(prioritizations_config.keys())))
6797        log.debug("Profiles to check: " + str(list(profiles)))
6798
6799        # Variables
6800        if table is not None:
6801            table_variants = table
6802        else:
6803            table_variants = self.get_table_variants(clause="update")
6804        log.debug(f"Table to prioritize: {table_variants}")
6805
6806        # Added columns
6807        added_columns = []
6808
6809        # Create list of PZfields
6810        # List of PZFields
6811        list_of_pzfields_original = pzfields + [
6812            pzfield + pzfields_sep + profile
6813            for pzfield in pzfields
6814            for profile in profiles
6815        ]
6816        list_of_pzfields = []
6817        log.debug(f"{list_of_pzfields_original}")
6818
6819        # Remove existing PZfields to use if exists
6820        for pzfield in list_of_pzfields_original:
6821            if self.get_header().infos.get(pzfield, None) is None:
6822                list_of_pzfields.append(pzfield)
6823                log.debug(f"VCF Input - Header - PZfield '{pzfield}' not in VCF")
6824            else:
6825                log.debug(f"VCF Input - Header - PZfield '{pzfield}' already in VCF")
6826
6827        if list_of_pzfields:
6828
6829            # Explode Infos prefix
6830            explode_infos_prefix = self.get_explode_infos_prefix()
6831
6832            # PZfields tags description
6833            PZfields_INFOS = {
6834                f"{pz_prefix}Tags": {
6835                    "ID": f"{pz_prefix}Tags",
6836                    "Number": ".",
6837                    "Type": "String",
6838                    "Description": "Variant tags based on annotation criteria",
6839                },
6840                f"{pz_prefix}Score": {
6841                    "ID": f"{pz_prefix}Score",
6842                    "Number": 1,
6843                    "Type": "Integer",
6844                    "Description": "Variant score based on annotation criteria",
6845                },
6846                f"{pz_prefix}Flag": {
6847                    "ID": f"{pz_prefix}Flag",
6848                    "Number": 1,
6849                    "Type": "String",
6850                    "Description": "Variant flag based on annotation criteria",
6851                },
6852                f"{pz_prefix}Comment": {
6853                    "ID": f"{pz_prefix}Comment",
6854                    "Number": ".",
6855                    "Type": "String",
6856                    "Description": "Variant comment based on annotation criteria",
6857                },
6858                f"{pz_prefix}Infos": {
6859                    "ID": f"{pz_prefix}Infos",
6860                    "Number": ".",
6861                    "Type": "String",
6862                    "Description": "Variant infos based on annotation criteria",
6863                },
6864                f"{pz_prefix}Class": {
6865                    "ID": f"{pz_prefix}Class",
6866                    "Number": ".",
6867                    "Type": "String",
6868                    "Description": "Variant class based on annotation criteria",
6869                },
6870            }
6871
6872            # Create INFO fields if not exist
6873            for field in PZfields_INFOS:
6874                field_ID = PZfields_INFOS[field]["ID"]
6875                field_description = PZfields_INFOS[field]["Description"]
6876                if field_ID not in self.get_header().infos and field_ID in pzfields:
6877                    field_description = (
6878                        PZfields_INFOS[field]["Description"]
6879                        + f", profile {default_profile}"
6880                    )
6881                    self.get_header().infos[field_ID] = vcf.parser._Info(
6882                        field_ID,
6883                        PZfields_INFOS[field]["Number"],
6884                        PZfields_INFOS[field]["Type"],
6885                        field_description,
6886                        "unknown",
6887                        "unknown",
6888                        code_type_map[PZfields_INFOS[field]["Type"]],
6889                    )
6890
6891            # Create INFO fields if not exist for each profile
6892            for profile in prioritizations_config:
6893                if profile in profiles or profiles == []:
6894                    for field in PZfields_INFOS:
6895                        field_ID = PZfields_INFOS[field]["ID"] + pzfields_sep + profile
6896                        field_description = (
6897                            PZfields_INFOS[field]["Description"]
6898                            + f", profile {profile}"
6899                        )
6900                        if (
6901                            field_ID not in self.get_header().infos
6902                            and field in pzfields
6903                        ):
6904                            self.get_header().infos[field_ID] = vcf.parser._Info(
6905                                field_ID,
6906                                PZfields_INFOS[field]["Number"],
6907                                PZfields_INFOS[field]["Type"],
6908                                field_description,
6909                                "unknown",
6910                                "unknown",
6911                                code_type_map[PZfields_INFOS[field]["Type"]],
6912                            )
6913
6914            # Header
6915            for pzfield in list_of_pzfields:
6916                if re.match(f"{pz_prefix}Score.*", pzfield):
6917                    added_column = self.add_column(
6918                        table_name=table_variants,
6919                        column_name=pzfield,
6920                        column_type="INTEGER",
6921                        default_value="0",
6922                    )
6923                elif re.match(f"{pz_prefix}Flag.*", pzfield):
6924                    added_column = self.add_column(
6925                        table_name=table_variants,
6926                        column_name=pzfield,
6927                        column_type="BOOLEAN",
6928                        default_value="1",
6929                    )
6930                elif re.match(f"{pz_prefix}Class.*", pzfield):
6931                    added_column = self.add_column(
6932                        table_name=table_variants,
6933                        column_name=pzfield,
6934                        column_type="VARCHAR[]",
6935                        default_value="null",
6936                    )
6937                else:
6938                    added_column = self.add_column(
6939                        table_name=table_variants,
6940                        column_name=pzfield,
6941                        column_type="STRING",
6942                        default_value="''",
6943                    )
6944                added_columns.append(added_column)
6945
6946            # Profiles
6947            if profiles:
6948
6949                # foreach profile in configuration file
6950                for profile in prioritizations_config:
6951
6952                    # If profile is asked in param, or ALL are asked (empty profile [])
6953                    if profile in profiles or profiles == []:
6954                        log.info(f"Profile '{profile}'")
6955
6956                        sql_set_info_option = ""
6957
6958                        sql_set_info = []
6959
6960                        # PZ fields set
6961
6962                        # PZScore
6963                        if (
6964                            f"{pz_prefix}Score{pzfields_sep}{profile}"
6965                            in list_of_pzfields
6966                        ):
6967                            sql_set_info.append(
6968                                f"""
6969                                    concat(
6970                                        '{pz_prefix}Score{pzfields_sep}{profile}=',
6971                                        {pz_prefix}Score{pzfields_sep}{profile}
6972                                    ) 
6973                                """
6974                            )
6975                            if (
6976                                profile == default_profile
6977                                and f"{pz_prefix}Score" in list_of_pzfields
6978                            ):
6979                                sql_set_info.append(
6980                                    f"""
6981                                        concat(
6982                                            '{pz_prefix}Score=',
6983                                            {pz_prefix}Score{pzfields_sep}{profile}
6984                                        )
6985                                    """
6986                                )
6987
6988                        # PZFlag
6989                        if (
6990                            f"{pz_prefix}Flag{pzfields_sep}{profile}"
6991                            in list_of_pzfields
6992                        ):
6993                            sql_set_info.append(
6994                                f"""
6995                                    concat(
6996                                        '{pz_prefix}Flag{pzfields_sep}{profile}=',
6997                                        CASE 
6998                                            WHEN {pz_prefix}Flag{pzfields_sep}{profile}==1
6999                                            THEN 'PASS'
7000                                            WHEN {pz_prefix}Flag{pzfields_sep}{profile}==0
7001                                            THEN 'FILTERED'
7002                                        END
7003                                    ) 
7004                                """
7005                            )
7006                            if (
7007                                profile == default_profile
7008                                and f"{pz_prefix}Flag" in list_of_pzfields
7009                            ):
7010                                sql_set_info.append(
7011                                    f"""
7012                                        concat(
7013                                            '{pz_prefix}Flag=',
7014                                            CASE 
7015                                                WHEN {pz_prefix}Flag{pzfields_sep}{profile}==1
7016                                                THEN 'PASS'
7017                                                WHEN {pz_prefix}Flag{pzfields_sep}{profile}==0
7018                                                THEN 'FILTERED'
7019                                            END
7020                                        )
7021                                    """
7022                                )
7023
7024                        # PZClass
7025                        if (
7026                            f"{pz_prefix}Class{pzfields_sep}{profile}"
7027                            in list_of_pzfields
7028                        ):
7029                            sql_set_info.append(
7030                                f"""
7031                                    concat(
7032                                        '{pz_prefix}Class{pzfields_sep}{profile}=',
7033                                        CASE
7034                                            WHEN len({pz_prefix}Class{pzfields_sep}{profile}) > 0
7035                                            THEN list_aggregate(list_distinct({pz_prefix}Class{pzfields_sep}{profile}), 'string_agg', ',')
7036                                            ELSE '.'
7037                                        END 
7038                                    )
7039                                    
7040                                """
7041                            )
7042                            if (
7043                                profile == default_profile
7044                                and f"{pz_prefix}Class" in list_of_pzfields
7045                            ):
7046                                sql_set_info.append(
7047                                    f"""
7048                                        concat(
7049                                            '{pz_prefix}Class=',
7050                                            CASE
7051                                                WHEN len({pz_prefix}Class{pzfields_sep}{profile}) > 0
7052                                                THEN list_aggregate(list_distinct({pz_prefix}Class{pzfields_sep}{profile}), 'string_agg', ',')
7053                                                ELSE '.'
7054                                            END 
7055                                        )
7056                                    """
7057                                )
7058
7059                        # PZComment
7060                        if (
7061                            f"{pz_prefix}Comment{pzfields_sep}{profile}"
7062                            in list_of_pzfields
7063                        ):
7064                            sql_set_info.append(
7065                                f"""
7066                                    CASE
7067                                        WHEN {pz_prefix}Comment{pzfields_sep}{profile} NOT IN ('')
7068                                        THEN concat('{pz_prefix}Comment{pzfields_sep}{profile}=', {pz_prefix}Comment{pzfields_sep}{profile})
7069                                        ELSE ''
7070                                    END
7071                                """
7072                            )
7073                            if (
7074                                profile == default_profile
7075                                and f"{pz_prefix}Comment" in list_of_pzfields
7076                            ):
7077                                sql_set_info.append(
7078                                    f"""
7079                                        CASE
7080                                            WHEN {pz_prefix}Comment{pzfields_sep}{profile} NOT IN ('')
7081                                            THEN concat('{pz_prefix}Comment=', {pz_prefix}Comment{pzfields_sep}{profile})
7082                                            ELSE ''
7083                                        END
7084                                    """
7085                                )
7086
7087                        # PZInfos
7088                        if (
7089                            f"{pz_prefix}Infos{pzfields_sep}{profile}"
7090                            in list_of_pzfields
7091                        ):
7092                            sql_set_info.append(
7093                                f"""
7094                                    CASE
7095                                        WHEN {pz_prefix}Infos{pzfields_sep}{profile} NOT IN ('')
7096                                        THEN concat('{pz_prefix}Infos{pzfields_sep}{profile}=', {pz_prefix}Infos{pzfields_sep}{profile})
7097                                        ELSE ''
7098                                    END
7099                                """
7100                            )
7101                            if (
7102                                profile == default_profile
7103                                and f"{pz_prefix}Infos" in list_of_pzfields
7104                            ):
7105                                sql_set_info.append(
7106                                    f"""
7107                                        CASE
7108                                            WHEN {pz_prefix}Infos{pzfields_sep}{profile} NOT IN ('')
7109                                            THEN concat('{pz_prefix}Infos=', {pz_prefix}Infos{pzfields_sep}{profile})
7110                                            ELSE ''
7111                                        END
7112                                    """
7113                                )
7114
7115                        # Merge PZfields
7116                        sql_set_info_option = ""
7117                        sql_set_sep = ""
7118                        for sql_set in sql_set_info:
7119                            if sql_set_sep:
7120                                sql_set_info_option += f"""
7121                                    , concat('{sql_set_sep}', {sql_set})
7122                                """
7123                            else:
7124                                sql_set_info_option += f"""
7125                                    , {sql_set}
7126                                """
7127                            sql_set_sep = ";"
7128
7129                        sql_queries = []
7130                        for annotation in prioritizations_config[profile]:
7131
7132                            # skip special sections
7133                            if annotation.startswith("_"):
7134                                continue
7135
7136                            # For each criterions
7137                            for criterion in prioritizations_config[profile][
7138                                annotation
7139                            ]:
7140
7141                                # Criterion mode
7142                                criterion_mode = None
7143                                if np.any(
7144                                    np.isin(list(criterion.keys()), ["type", "value"])
7145                                ):
7146                                    criterion_mode = "operation"
7147                                elif np.any(
7148                                    np.isin(list(criterion.keys()), ["sql", "fields"])
7149                                ):
7150                                    criterion_mode = "sql"
7151                                log.debug(f"Criterion Mode: {criterion_mode}")
7152
7153                                # Criterion parameters
7154                                criterion_type = criterion.get("type", None)
7155                                criterion_value = criterion.get("value", None)
7156                                criterion_sql = criterion.get("sql", None)
7157                                criterion_fields = criterion.get("fields", None)
7158                                criterion_score = criterion.get("score", 0)
7159                                criterion_flag = criterion.get("flag", "PASS")
7160                                criterion_class = criterion.get("class", None)
7161                                criterion_flag_bool = criterion_flag == "PASS"
7162                                criterion_comment = (
7163                                    ", ".join(criterion.get("comment", []))
7164                                    .replace("'", "''")
7165                                    .replace(";", ",")
7166                                    .replace("\t", " ")
7167                                )
7168                                criterion_infos = (
7169                                    str(criterion)
7170                                    .replace("'", "''")
7171                                    .replace(";", ",")
7172                                    .replace("\t", " ")
7173                                )
7174
7175                                # SQL
7176                                if criterion_sql is not None and isinstance(
7177                                    criterion_sql, list
7178                                ):
7179                                    criterion_sql = " ".join(criterion_sql)
7180
7181                                # Fields and explode
7182                                if criterion_fields is None:
7183                                    criterion_fields = [annotation]
7184                                if not isinstance(criterion_fields, list):
7185                                    criterion_fields = str(criterion_fields).split(",")
7186
7187                                # Class
7188                                if criterion_class is not None and not isinstance(
7189                                    criterion_class, list
7190                                ):
7191                                    criterion_class = str(criterion_class).split(",")
7192
7193                                for annotation_field in criterion_fields:
7194
7195                                    # Explode specific annotation
7196                                    log.debug(
7197                                        f"Explode annotation '{annotation_field}'"
7198                                    )
7199                                    added_columns += self.explode_infos(
7200                                        prefix=explode_infos_prefix,
7201                                        fields=[annotation_field],
7202                                        table=table_variants,
7203                                    )
7204                                    extra_infos = self.get_extra_infos(
7205                                        table=table_variants
7206                                    )
7207
7208                                    # Check if annotation field is present
7209                                    if (
7210                                        f"{explode_infos_prefix}{annotation_field}"
7211                                        not in extra_infos
7212                                    ):
7213                                        msq_err = f"Annotation '{annotation_field}' not in data"
7214                                        log.error(msq_err)
7215                                        raise ValueError(msq_err)
7216                                    else:
7217                                        log.debug(
7218                                            f"Annotation '{annotation_field}' in data"
7219                                        )
7220
7221                                sql_set = []
7222                                sql_set_info = []
7223
7224                                # PZ fields set
7225
7226                                # PZScore
7227                                if (
7228                                    f"{pz_prefix}Score{pzfields_sep}{profile}"
7229                                    in list_of_pzfields
7230                                ):
7231                                    # if prioritization_score_mode == "HOWARD":
7232                                    #     sql_set.append(
7233                                    #         f"{pz_prefix}Score{pzfields_sep}{profile} = {pz_prefix}Score{pzfields_sep}{profile} + {criterion_score}"
7234                                    #     )
7235                                    # VaRank prioritization score mode
7236                                    if prioritization_score_mode == "VaRank":
7237                                        sql_set.append(
7238                                            f"{pz_prefix}Score{pzfields_sep}{profile} = CASE WHEN {criterion_score}>{pz_prefix}Score{pzfields_sep}{profile} THEN {criterion_score} END"
7239                                        )
7240                                    # default HOWARD prioritization score mode
7241                                    else:
7242                                        sql_set.append(
7243                                            f"{pz_prefix}Score{pzfields_sep}{profile} = {pz_prefix}Score{pzfields_sep}{profile} + {criterion_score}"
7244                                        )
7245
7246                                # PZFlag
7247                                if (
7248                                    f"{pz_prefix}Flag{pzfields_sep}{profile}"
7249                                    in list_of_pzfields
7250                                ):
7251                                    sql_set.append(
7252                                        f"{pz_prefix}Flag{pzfields_sep}{profile} = {pz_prefix}Flag{pzfields_sep}{profile} AND {criterion_flag_bool}"
7253                                    )
7254
7255                                # PZClass
7256                                if (
7257                                    f"{pz_prefix}Class{pzfields_sep}{profile}"
7258                                    in list_of_pzfields
7259                                    and criterion_class is not None
7260                                ):
7261                                    sql_set.append(
7262                                        f" {pz_prefix}Class{pzfields_sep}{profile} = list_concat(list_distinct({pz_prefix}Class{pzfields_sep}{profile}), {criterion_class}) "
7263                                    )
7264
7265                                # PZComment
7266                                if (
7267                                    f"{pz_prefix}Comment{pzfields_sep}{profile}"
7268                                    in list_of_pzfields
7269                                ):
7270                                    sql_set.append(
7271                                        f"""
7272                                            {pz_prefix}Comment{pzfields_sep}{profile} = 
7273                                                concat(
7274                                                    {pz_prefix}Comment{pzfields_sep}{profile},
7275                                                    CASE 
7276                                                        WHEN {pz_prefix}Comment{pzfields_sep}{profile}!=''
7277                                                        THEN ', '
7278                                                        ELSE ''
7279                                                    END,
7280                                                    '{criterion_comment}'
7281                                                )
7282                                        """
7283                                    )
7284
7285                                # PZInfos
7286                                if (
7287                                    f"{pz_prefix}Infos{pzfields_sep}{profile}"
7288                                    in list_of_pzfields
7289                                ):
7290                                    sql_set.append(
7291                                        f"""
7292                                            {pz_prefix}Infos{pzfields_sep}{profile} = 
7293                                                concat(
7294                                                    {pz_prefix}Infos{pzfields_sep}{profile},
7295                                                    '{criterion_infos}'
7296                                                )
7297                                        """
7298                                    )
7299                                sql_set_option = ",".join(sql_set)
7300
7301                                # Criterion and comparison
7302                                if sql_set_option:
7303
7304                                    if criterion_mode in ["operation"]:
7305
7306                                        try:
7307                                            float(criterion_value)
7308                                            sql_update = f"""
7309                                                UPDATE {table_variants}
7310                                                SET {sql_set_option}
7311                                                WHERE CAST("{explode_infos_prefix}{annotation}" AS VARCHAR) NOT IN ('','.')
7312                                                AND CAST("{explode_infos_prefix}{annotation}" AS FLOAT){comparison_map[criterion_type]}{criterion_value}
7313                                            """
7314                                        except:
7315                                            contains_option = ""
7316                                            if criterion_type == "contains":
7317                                                contains_option = ".*"
7318                                            sql_update = f"""
7319                                                UPDATE {table_variants}
7320                                                SET {sql_set_option}
7321                                                WHERE "{explode_infos_prefix}{annotation}" SIMILAR TO '{contains_option}{criterion_value}{contains_option}'
7322                                            """
7323                                        sql_queries.append(sql_update)
7324
7325                                    elif criterion_mode in ["sql"]:
7326
7327                                        sql_update = f"""
7328                                            UPDATE {table_variants}
7329                                            SET {sql_set_option}
7330                                            WHERE {criterion_sql}
7331                                        """
7332                                        sql_queries.append(sql_update)
7333
7334                                    else:
7335                                        msg_err = f"Prioritization criterion mode failed (either 'operation' or 'sql')"
7336                                        log.error(msg_err)
7337                                        raise ValueError(msg_err)
7338
7339                                else:
7340                                    log.warning(
7341                                        f"NO SQL SET option for '{annotation}' - '{criterion}'"
7342                                    )
7343
7344                        # PZTags
7345                        if (
7346                            f"{pz_prefix}Tags{pzfields_sep}{profile}"
7347                            in list_of_pzfields
7348                        ):
7349
7350                            # Create PZFalgs value
7351                            pztags_value = ""
7352                            pztags_sep_default = ","
7353                            pztags_sep = ""
7354                            for pzfield in pzfields:
7355                                if pzfield not in [f"{pz_prefix}Tags"]:
7356                                    if (
7357                                        f"{pzfield}{pzfields_sep}{profile}"
7358                                        in list_of_pzfields
7359                                    ):
7360                                        if pzfield in [f"{pz_prefix}Flag"]:
7361                                            pztags_value += f"""{pztags_sep}{pzfield}#', 
7362                                                CASE WHEN {pz_prefix}Flag{pzfields_sep}{profile}
7363                                                    THEN 'PASS'
7364                                                    ELSE 'FILTERED'
7365                                                END, '"""
7366                                        elif pzfield in [f"{pz_prefix}Class"]:
7367                                            pztags_value += f"""{pztags_sep}{pzfield}#', 
7368                                                CASE WHEN len({pz_prefix}Class{pzfields_sep}{profile}) > 0
7369                                                    THEN list_aggregate(list_distinct({pz_prefix}Class{pzfields_sep}{profile}), 'string_agg', ',')
7370                                                    ELSE '.'
7371                                                END, '"""
7372                                        else:
7373                                            pztags_value += f"{pztags_sep}{pzfield}#', {pzfield}{pzfields_sep}{profile}, '"
7374                                        pztags_sep = pztags_sep_default
7375
7376                            # Add Query update for PZFlags
7377                            sql_update_pztags = f"""
7378                                UPDATE {table_variants}
7379                                SET INFO = concat(
7380                                        INFO,
7381                                        CASE WHEN INFO NOT in ('','.')
7382                                                THEN ';'
7383                                                ELSE ''
7384                                        END,
7385                                        '{pz_prefix}Tags{pzfields_sep}{profile}={pztags_value}'
7386                                    )
7387                                """
7388                            sql_queries.append(sql_update_pztags)
7389
7390                            # Add Query update for PZFlags for default
7391                            if profile == default_profile:
7392                                sql_update_pztags_default = f"""
7393                                UPDATE {table_variants}
7394                                SET INFO = concat(
7395                                        INFO,
7396                                        ';',
7397                                        '{pz_prefix}Tags={pztags_value}'
7398                                    )
7399                                """
7400                                sql_queries.append(sql_update_pztags_default)
7401
7402                        log.info(f"""Profile '{profile}' - Prioritization... """)
7403
7404                        if sql_queries:
7405
7406                            for sql_query in sql_queries:
7407                                log.debug(
7408                                    f"""Profile '{profile}' - Prioritization query: {sql_query}... """
7409                                )
7410                                self.conn.execute(sql_query)
7411
7412                        log.info(f"""Profile '{profile}' - Update... """)
7413                        sql_query_update = f"""
7414                            UPDATE {table_variants}
7415                            SET INFO =  
7416                                concat(
7417                                    CASE
7418                                        WHEN INFO NOT IN ('','.')
7419                                        THEN concat(INFO, ';')
7420                                        ELSE ''
7421                                    END
7422                                    {sql_set_info_option}
7423                                )
7424                        """
7425                        self.conn.execute(sql_query_update)
7426
7427        else:
7428
7429            log.warning(f"No profiles in parameters")
7430
7431        # Remove added columns
7432        for added_column in added_columns:
7433            self.drop_column(column=added_column)
7434
7435        # Explode INFOS fields into table fields
7436        if self.get_explode_infos():
7437            self.explode_infos(
7438                prefix=self.get_explode_infos_prefix(),
7439                fields=self.get_explode_infos_fields(),
7440                force=True,
7441            )
7442
7443        return True

The prioritization function in Python processes VCF files, adds new INFO fields, and prioritizes variants based on configured profiles and criteria.

Parameters

table: The table parameter in the prioritization function is used to specify the name of the table (presumably a VCF file) on which the prioritization operation will be performed. If a table name is provided, the method will prioritize the variants in that specific table
pz_prefix: The pz_prefix parameter is used to specify a prefix that will be added to certain INFO fields in a VCF file during the prioritization process. If this parameter is not provided, the code will use a default prefix value of "PZ"
pz_param: The pz_param parameter in the prioritization method is used to pass additional parameters specific to the prioritization process. These parameters can include settings related to prioritization profiles, fields, scoring modes, flags, comments, and other configurations needed for the prioritization of variants in a V

Returns

A boolean value (True) is being returned from the prioritization function.

def annotation_hgvs(self, threads: int = None) -> None: View Source

7449    def annotation_hgvs(self, threads: int = None) -> None:
7450        """
7451        The `annotation_hgvs` function performs HGVS annotation on a set of variants using genomic
7452        coordinates and alleles.
7453
7454        :param threads: The `threads` parameter is an optional integer that specifies the number of
7455        threads to use for parallel processing. If no value is provided, it will default to the number
7456        of threads obtained from the `get_threads()` method
7457        :type threads: int
7458        """
7459
7460        # Function for each partition of the Dask Dataframe
7461        def partition_function(partition):
7462            """
7463            The function `partition_function` applies the `annotation_hgvs_partition` function to
7464            each row of a DataFrame called `partition`.
7465
7466            :param partition: The parameter "partition" is a pandas DataFrame that contains the data
7467            to be processed
7468            :return: the result of applying the "annotation_hgvs_partition" function to each row of
7469            the "partition" dataframe along the axis 1.
7470            """
7471            return partition.apply(annotation_hgvs_partition, axis=1)
7472
7473        def annotation_hgvs_partition(row) -> str:
7474            """
7475            The function `annotation_hgvs_partition` takes in a row of data and returns a string
7476            containing a list of HGVS names associated with the given genomic coordinates and alleles.
7477
7478            :param row: A dictionary-like object that contains the values for the following keys:
7479            :return: a string that contains the HGVS names associated with the given row of data.
7480            """
7481
7482            chr = row["CHROM"]
7483            pos = row["POS"]
7484            ref = row["REF"]
7485            alt = row["ALT"]
7486
7487            # Find list of associated transcripts
7488            transcripts_list = list(
7489                polars_conn.execute(
7490                    f"""
7491                SELECT transcript
7492                FROM refseq_df
7493                WHERE CHROM='{chr}'
7494                AND POS={pos}
7495            """
7496                )["transcript"]
7497            )
7498
7499            # Full HGVS annotation in list
7500            hgvs_full_list = []
7501
7502            for transcript_name in transcripts_list:
7503
7504                # Transcript
7505                transcript = get_transcript(
7506                    transcripts=transcripts, transcript_name=transcript_name
7507                )
7508                # Exon
7509                if use_exon:
7510                    exon = transcript.find_exon_number(pos)
7511                else:
7512                    exon = None
7513                # Protein
7514                transcript_protein = None
7515                if use_protein or add_protein or full_format:
7516                    transcripts_protein = list(
7517                        polars_conn.execute(
7518                            f"""
7519                        SELECT protein
7520                        FROM refseqlink_df
7521                        WHERE transcript='{transcript_name}'
7522                        LIMIT 1
7523                    """
7524                        )["protein"]
7525                    )
7526                    if len(transcripts_protein):
7527                        transcript_protein = transcripts_protein[0]
7528
7529                # HGVS name
7530                hgvs_name = format_hgvs_name(
7531                    chr,
7532                    pos,
7533                    ref,
7534                    alt,
7535                    genome=genome,
7536                    transcript=transcript,
7537                    transcript_protein=transcript_protein,
7538                    exon=exon,
7539                    use_gene=use_gene,
7540                    use_protein=use_protein,
7541                    full_format=full_format,
7542                    use_version=use_version,
7543                    codon_type=codon_type,
7544                )
7545                hgvs_full_list.append(hgvs_name)
7546                if add_protein and not use_protein and not full_format:
7547                    hgvs_name = format_hgvs_name(
7548                        chr,
7549                        pos,
7550                        ref,
7551                        alt,
7552                        genome=genome,
7553                        transcript=transcript,
7554                        transcript_protein=transcript_protein,
7555                        exon=exon,
7556                        use_gene=use_gene,
7557                        use_protein=True,
7558                        full_format=False,
7559                        use_version=use_version,
7560                        codon_type=codon_type,
7561                    )
7562                    hgvs_full_list.append(hgvs_name)
7563
7564            # Create liste of HGVS annotations
7565            hgvs_full = ",".join(hgvs_full_list)
7566
7567            return hgvs_full
7568
7569        # Polars connexion
7570        polars_conn = pl.SQLContext(register_globals=True, eager=True)
7571
7572        # Config
7573        config = self.get_config()
7574
7575        # Databases
7576        # Genome
7577        databases_genomes_folders = (
7578            config.get("folders", {})
7579            .get("databases", {})
7580            .get("genomes", DEFAULT_GENOME_FOLDER)
7581        )
7582        databases_genome = (
7583            config.get("folders", {}).get("databases", {}).get("genomes", "")
7584        )
7585        # refseq database folder
7586        databases_refseq_folders = (
7587            config.get("folders", {})
7588            .get("databases", {})
7589            .get("refseq", DEFAULT_REFSEQ_FOLDER)
7590        )
7591        # refseq
7592        databases_refseq = config.get("databases", {}).get("refSeq", None)
7593        # refSeqLink
7594        databases_refseqlink = config.get("databases", {}).get("refSeqLink", None)
7595
7596        # Param
7597        param = self.get_param()
7598
7599        # Quick HGVS
7600        if "hgvs_options" in param and param.get("hgvs_options", ""):
7601            log.info(f"Quick HGVS Annotation:")
7602            if not param.get("hgvs", None):
7603                param["hgvs"] = {}
7604            for option in param.get("hgvs_options", "").split(","):
7605                option_var_val = option.split("=")
7606                option_var = option_var_val[0]
7607                if len(option_var_val) > 1:
7608                    option_val = option_var_val[1]
7609                else:
7610                    option_val = "True"
7611                if option_val.upper() in ["TRUE"]:
7612                    option_val = True
7613                elif option_val.upper() in ["FALSE"]:
7614                    option_val = False
7615                log.info(f"   {option_var}={option_val}")
7616                param["hgvs"][option_var] = option_val
7617
7618        # Check if HGVS annotation enabled
7619        if "hgvs" in param:
7620            log.info(f"HGVS Annotation... ")
7621            for hgvs_option in param.get("hgvs", {}):
7622                log.info(f"{hgvs_option}: {param.get('hgvs',{}).get(hgvs_option)}")
7623        else:
7624            return
7625
7626        # HGVS Param
7627        param_hgvs = param.get("hgvs", {})
7628        use_exon = param_hgvs.get("use_exon", False)
7629        use_gene = param_hgvs.get("use_gene", False)
7630        use_protein = param_hgvs.get("use_protein", False)
7631        add_protein = param_hgvs.get("add_protein", False)
7632        full_format = param_hgvs.get("full_format", False)
7633        use_version = param_hgvs.get("use_version", False)
7634        codon_type = param_hgvs.get("codon_type", "3")
7635
7636        # refSseq refSeqLink
7637        databases_refseq = param_hgvs.get("refseq", databases_refseq)
7638        databases_refseqlink = param_hgvs.get("refseqlink", databases_refseqlink)
7639
7640        # Assembly
7641        assembly = param.get("assembly", config.get("assembly", DEFAULT_ASSEMBLY))
7642
7643        # Genome
7644        genome_file = None
7645        if find_genome(databases_genome):
7646            genome_file = find_genome(databases_genome)
7647        else:
7648            genome_file = find_genome(
7649                genome_path=databases_genomes_folders, assembly=assembly
7650            )
7651        log.debug("Genome: " + str(genome_file))
7652
7653        # refSseq
7654        refseq_file = find_file_prefix(
7655            input_file=databases_refseq,
7656            prefix="ncbiRefSeq",
7657            folder=databases_refseq_folders,
7658            assembly=assembly,
7659        )
7660        log.debug("refSeq: " + str(refseq_file))
7661
7662        # refSeqLink
7663        refseqlink_file = find_file_prefix(
7664            input_file=databases_refseqlink,
7665            prefix="ncbiRefSeqLink",
7666            folder=databases_refseq_folders,
7667            assembly=assembly,
7668        )
7669        log.debug("refSeqLink: " + str(refseqlink_file))
7670
7671        # Threads
7672        if not threads:
7673            threads = self.get_threads()
7674        log.debug("Threads: " + str(threads))
7675
7676        # Variables
7677        table_variants = self.get_table_variants(clause="update")
7678
7679        # Get variants SNV and InDel only
7680        query_variants = f"""
7681            SELECT "#CHROM" AS CHROM, POS, REF, ALT
7682            FROM {table_variants}
7683            WHERE REF ~ '^[A-Za-z]+$' AND ALT ~ '^[A-Za-z]+$'
7684            """
7685        df_variants = self.get_query_to_df(query_variants)
7686
7687        # Added columns
7688        added_columns = []
7689
7690        # Add hgvs column in variants table
7691        hgvs_column_name = "hgvs_" + str(random.randrange(1000))
7692        added_column = self.add_column(
7693            table_variants, hgvs_column_name, "STRING", default_value=None
7694        )
7695        added_columns.append(added_column)
7696
7697        log.debug(f"refSeq loading...")
7698        # refSeq in duckDB
7699        refseq_table = get_refseq_table(
7700            conn=self.conn, refseq_table="refseq", refseq_file=refseq_file
7701        )
7702        # Loading all refSeq in Dataframe
7703        refseq_query = f"""
7704            SELECT df_variants.CHROM, df_variants.POS, {refseq_table}.name AS transcript
7705            FROM {refseq_table}
7706            JOIN df_variants ON (
7707                {refseq_table}.chrom = df_variants.CHROM
7708                AND {refseq_table}.txStart<=df_variants.POS
7709                AND {refseq_table}.txEnd>=df_variants.POS
7710            )
7711        """
7712        refseq_df = self.conn.query(refseq_query).pl()
7713
7714        if refseqlink_file:
7715            log.debug(f"refSeqLink loading...")
7716            # refSeqLink in duckDB
7717            refseqlink_table = get_refseq_table(
7718                conn=self.conn, refseq_table="refseqlink", refseq_file=refseqlink_file
7719            )
7720            # Loading all refSeqLink in Dataframe
7721            protacc_column = "protAcc_with_ver"
7722            mrnaacc_column = "mrnaAcc_with_ver"
7723            refseqlink_query = f"""
7724                SELECT {refseq_table}.chrom, {protacc_column} AS protein, {mrnaacc_column} AS transcript
7725                FROM {refseqlink_table} 
7726                JOIN {refseq_table} ON ({refseq_table}.name = {refseqlink_table}.mrnaAcc_with_ver)
7727                WHERE protAcc_without_ver IS NOT NULL
7728            """
7729            # Polars Dataframe
7730            refseqlink_df = self.conn.query(f"{refseqlink_query}").pl()
7731
7732        # Read RefSeq transcripts into a python dict/model.
7733        log.debug(f"Transcripts loading...")
7734        with tempfile.TemporaryDirectory() as tmpdir:
7735            transcripts_query = f"""
7736                COPY (
7737                    SELECT {refseq_table}.*
7738                    FROM {refseq_table}
7739                    JOIN df_variants ON (
7740                        {refseq_table}.chrom=df_variants.CHROM
7741                        AND {refseq_table}.txStart<=df_variants.POS
7742                        AND {refseq_table}.txEnd>=df_variants.POS
7743                    )
7744                )
7745                TO '{tmpdir}/transcript.tsv' (DELIMITER '\t');
7746            """
7747            self.conn.query(transcripts_query)
7748            with open(f"{tmpdir}/transcript.tsv") as infile:
7749                transcripts = read_transcripts(infile)
7750
7751        # Polars connexion
7752        polars_conn = pl.SQLContext(register_globals=True, eager=True)
7753
7754        log.debug("Genome loading...")
7755        # Read genome sequence using pyfaidx.
7756        genome = Fasta(genome_file)
7757
7758        log.debug("Start annotation HGVS...")
7759
7760        # Create
7761        # a Dask Dataframe from Pandas dataframe with partition as number of threads
7762        ddf = dd.from_pandas(df_variants, npartitions=threads)
7763
7764        # Use dask.dataframe.apply() to apply function on each partition
7765        ddf[hgvs_column_name] = ddf.map_partitions(partition_function)
7766
7767        # Convert Dask DataFrame to Pandas Dataframe
7768        df = ddf.compute()
7769
7770        # Convert Pandas dataframe to parquet (due to error in cast VARCHAR -> NULL ???)
7771        with tempfile.TemporaryDirectory() as tmpdir:
7772            df_parquet = os.path.join(tmpdir, "df.parquet")
7773            df.to_parquet(df_parquet)
7774
7775            # Update hgvs column
7776            update_variant_query = f"""
7777                UPDATE {table_variants}
7778                SET "{hgvs_column_name}"=df."{hgvs_column_name}"
7779                FROM read_parquet('{df_parquet}') as df
7780                WHERE variants."#CHROM" = df.CHROM
7781                AND variants.POS = df.POS
7782                AND variants.REF = df.REF
7783                AND variants.ALT = df.ALT
7784                AND df."{hgvs_column_name}" NOT IN ('') AND df."{hgvs_column_name}" NOT NULL
7785                """
7786            self.execute_query(update_variant_query)
7787
7788        # Update INFO column
7789        sql_query_update = f"""
7790            UPDATE {table_variants}
7791            SET INFO = 
7792                concat(
7793                    CASE 
7794                        WHEN INFO NOT IN ('','.')
7795                        THEN concat(INFO, ';')
7796                        ELSE ''
7797                    END,
7798                    'hgvs=',
7799                    {hgvs_column_name}
7800                )
7801            WHERE "{hgvs_column_name}" NOT IN ('') AND "{hgvs_column_name}" NOT NULL
7802            """
7803        self.execute_query(sql_query_update)
7804
7805        # Add header
7806        HGVS_INFOS = {
7807            "hgvs": {
7808                "ID": "hgvs",
7809                "Number": ".",
7810                "Type": "String",
7811                "Description": f"HGVS annotatation with HOWARD",
7812            }
7813        }
7814
7815        for field in HGVS_INFOS:
7816            field_ID = HGVS_INFOS[field]["ID"]
7817            field_description = HGVS_INFOS[field]["Description"]
7818            self.get_header().infos[field_ID] = vcf.parser._Info(
7819                field_ID,
7820                HGVS_INFOS[field]["Number"],
7821                HGVS_INFOS[field]["Type"],
7822                field_description,
7823                "unknown",
7824                "unknown",
7825                code_type_map[HGVS_INFOS[field]["Type"]],
7826            )
7827
7828        # Remove added columns
7829        for added_column in added_columns:
7830            self.drop_column(column=added_column)

The annotation_hgvs function performs HGVS annotation on a set of variants using genomic coordinates and alleles.

Parameters

threads: The threads parameter is an optional integer that specifies the number of threads to use for parallel processing. If no value is provided, it will default to the number of threads obtained from the get_threads() method

def get_operations_help( self, operations_config_dict: dict = {}, operations_config_file: str = None) -> list: View Source

7836    def get_operations_help(
7837        self, operations_config_dict: dict = {}, operations_config_file: str = None
7838    ) -> list:
7839
7840        # Init
7841        operations_help = []
7842
7843        # operations
7844        operations = self.get_config_json(
7845            name="calculations",
7846            config_dict=operations_config_dict,
7847            config_file=operations_config_file,
7848        )
7849        for op in operations:
7850            op_name = operations[op].get("name", op).upper()
7851            op_description = operations[op].get("description", op_name)
7852            op_available = operations[op].get("available", False)
7853            if op_available:
7854                operations_help.append(f"   {op_name}: {op_description}")
7855
7856        # Sort operations
7857        operations_help.sort()
7858
7859        # insert header
7860        operations_help.insert(0, "Available calculation operations:")
7861
7862        # Return
7863        return operations_help

def calculation( self, operations: dict = {}, operations_config_dict: dict = {}, operations_config_file: str = None) -> None: View Source

7865    def calculation(
7866        self,
7867        operations: dict = {},
7868        operations_config_dict: dict = {},
7869        operations_config_file: str = None,
7870    ) -> None:
7871        """
7872        It takes a list of operations, and for each operation, it checks if it's a python or sql
7873        operation, and then calls the appropriate function
7874
7875        param json example:
7876            "calculation": {
7877                "NOMEN": {
7878                    "options": {
7879                        "hgvs_field": "hgvs"
7880                    },
7881                "middle" : null
7882            }
7883        """
7884
7885        # Param
7886        param = self.get_param()
7887
7888        # operations config
7889        operations_config = self.get_config_json(
7890            name="calculations",
7891            config_dict=operations_config_dict,
7892            config_file=operations_config_file,
7893        )
7894
7895        # Upper keys
7896        operations_config = {k.upper(): v for k, v in operations_config.items()}
7897
7898        # Calculations
7899
7900        # Operations from param
7901        operations = param.get("calculation", {}).get("calculations", operations)
7902
7903        # Quick calculation - add
7904        if param.get("calculations", None):
7905            calculations_list = [
7906                value for value in param.get("calculations", "").split(",")
7907            ]
7908            log.info(f"Quick Calculations:")
7909            for calculation_key in calculations_list:
7910                log.info(f"   {calculation_key}")
7911            for calculation_operation in calculations_list:
7912                if calculation_operation.upper() not in operations:
7913                    operations[calculation_operation.upper()] = {}
7914                    add_value_into_dict(
7915                        dict_tree=param,
7916                        sections=[
7917                            "calculation",
7918                            "calculations",
7919                            calculation_operation.upper(),
7920                        ],
7921                        value={},
7922                    )
7923
7924        # Operations for calculation
7925        if not operations:
7926            operations = param.get("calculation", {}).get("calculations", {})
7927
7928        if operations:
7929            log.info(f"Calculations...")
7930
7931        # For each operations
7932        for operation_name in operations:
7933            operation_name = operation_name.upper()
7934            if operation_name not in [""]:
7935                if operation_name in operations_config:
7936                    log.info(f"Calculation '{operation_name}'")
7937                    operation = operations_config[operation_name]
7938                    operation_type = operation.get("type", "sql")
7939                    if operation_type == "python":
7940                        self.calculation_process_function(
7941                            operation=operation, operation_name=operation_name
7942                        )
7943                    elif operation_type == "sql":
7944                        self.calculation_process_sql(
7945                            operation=operation, operation_name=operation_name
7946                        )
7947                    else:
7948                        log.error(
7949                            f"Operations config: Type '{operation_type}' NOT available"
7950                        )
7951                        raise ValueError(
7952                            f"Operations config: Type '{operation_type}' NOT available"
7953                        )
7954                else:
7955                    log.error(
7956                        f"Operations config: Calculation '{operation_name}' NOT available"
7957                    )
7958                    raise ValueError(
7959                        f"Operations config: Calculation '{operation_name}' NOT available"
7960                    )
7961
7962        # Explode INFOS fields into table fields
7963        if self.get_explode_infos():
7964            self.explode_infos(
7965                prefix=self.get_explode_infos_prefix(),
7966                fields=self.get_explode_infos_fields(),
7967                force=True,
7968            )

It takes a list of operations, and for each operation, it checks if it's a python or sql operation, and then calls the appropriate function

param json example: "calculation": { "NOMEN": { "options": { "hgvs_field": "hgvs" }, "middle" : null }

def calculation_process_sql(self, operation: dict, operation_name: str = 'unknown') -> None: View Source

7970    def calculation_process_sql(
7971        self, operation: dict, operation_name: str = "unknown"
7972    ) -> None:
7973        """
7974        The `calculation_process_sql` function takes in a mathematical operation as a string and
7975        performs the operation, updating the specified table with the result.
7976
7977        :param operation: The `operation` parameter is a dictionary that contains information about the
7978        mathematical operation to be performed. It includes the following keys:
7979        :type operation: dict
7980        :param operation_name: The `operation_name` parameter is a string that represents the name of
7981        the mathematical operation being performed. It is used for logging and error handling purposes,
7982        defaults to unknown
7983        :type operation_name: str (optional)
7984        """
7985
7986        # table variants
7987        table_variants = self.get_table_variants(clause="alter")
7988
7989        # Operation infos
7990        operation_name = operation.get("name", "unknown")
7991        log.debug(f"process sql {operation_name}")
7992        output_column_name = operation.get("output_column_name", operation_name)
7993        output_column_type = operation.get("output_column_type", "String")
7994        prefix = operation.get("explode_infos_prefix", "")
7995        output_column_type_sql = code_type_map_to_sql.get(output_column_type, "VARCHAR")
7996        output_column_description = operation.get(
7997            "output_column_description", f"{operation_name} operation"
7998        )
7999        operation_query = operation.get("operation_query", None)
8000        if isinstance(operation_query, list):
8001            operation_query = " ".join(operation_query)
8002        operation_info_fields = operation.get("info_fields", [])
8003        operation_info_fields_check = operation.get("info_fields_check", False)
8004        operation_info = operation.get("operation_info", True)
8005
8006        if operation_query:
8007
8008            # Info fields check
8009            operation_info_fields_check_result = True
8010            if operation_info_fields_check:
8011                header_infos = self.get_header().infos
8012                for info_field in operation_info_fields:
8013                    operation_info_fields_check_result = (
8014                        operation_info_fields_check_result
8015                        and info_field in header_infos
8016                    )
8017
8018            # If info fields available
8019            if operation_info_fields_check_result:
8020
8021                # Added_columns
8022                added_columns = []
8023
8024                # Create VCF header field
8025                vcf_reader = self.get_header()
8026                vcf_reader.infos[output_column_name] = vcf.parser._Info(
8027                    output_column_name,
8028                    ".",
8029                    output_column_type,
8030                    output_column_description,
8031                    "howard calculation",
8032                    "0",
8033                    self.code_type_map.get(output_column_type),
8034                )
8035
8036                # Explode infos if needed
8037                log.debug(f"calculation_process_sql prefix {prefix}")
8038                added_columns += self.explode_infos(
8039                    prefix=prefix,
8040                    fields=[output_column_name] + operation_info_fields,
8041                    force=True,
8042                )
8043
8044                # Create column
8045                added_column = self.add_column(
8046                    table_name=table_variants,
8047                    column_name=prefix + output_column_name,
8048                    column_type=output_column_type_sql,
8049                    default_value="null",
8050                )
8051                added_columns.append(added_column)
8052
8053                # Operation calculation
8054                try:
8055
8056                    # Query to update calculation column
8057                    sql_update = f"""
8058                        UPDATE {table_variants}
8059                        SET "{prefix}{output_column_name}" = ({operation_query})
8060                    """
8061                    self.conn.execute(sql_update)
8062
8063                    # Add to INFO
8064                    if operation_info:
8065                        sql_update_info = f"""
8066                            UPDATE {table_variants}
8067                            SET "INFO" =
8068                                concat(
8069                                    CASE
8070                                        WHEN "INFO" IS NOT NULL
8071                                        THEN concat("INFO", ';')
8072                                        ELSE ''
8073                                    END,
8074                                    '{output_column_name}=',
8075                                    "{prefix}{output_column_name}"
8076                                )
8077                            WHERE "{prefix}{output_column_name}" IS NOT NULL AND "{prefix}{output_column_name}" NOT IN ('')
8078                        """
8079                        self.conn.execute(sql_update_info)
8080
8081                except:
8082                    log.error(
8083                        f"Operations config: Calculation '{operation_name}' query failed"
8084                    )
8085                    raise ValueError(
8086                        f"Operations config: Calculation '{operation_name}' query failed"
8087                    )
8088
8089                # Remove added columns
8090                for added_column in added_columns:
8091                    log.debug(f"added_column: {added_column}")
8092                    self.drop_column(column=added_column)
8093
8094            else:
8095                log.error(
8096                    f"Operations config: Calculation '{operation_name}' DOES NOT contain all mandatory fields {operation_info_fields}"
8097                )
8098                raise ValueError(
8099                    f"Operations config: Calculation '{operation_name}' DOES NOT contain all mandatory fields {operation_info_fields}"
8100                )
8101
8102        else:
8103            log.error(
8104                f"Operations config: Calculation '{operation_name}' query NOT defined"
8105            )
8106            raise ValueError(
8107                f"Operations config: Calculation '{operation_name}' query NOT defined"
8108            )

The calculation_process_sql function takes in a mathematical operation as a string and performs the operation, updating the specified table with the result.

Parameters

operation: The operation parameter is a dictionary that contains information about the mathematical operation to be performed. It includes the following keys:
operation_name: The operation_name parameter is a string that represents the name of the mathematical operation being performed. It is used for logging and error handling purposes, defaults to unknown

def calculation_process_function(self, operation: dict, operation_name: str = 'unknown') -> None: View Source

8110    def calculation_process_function(
8111        self, operation: dict, operation_name: str = "unknown"
8112    ) -> None:
8113        """
8114        The `calculation_process_function` takes in an operation dictionary and performs the specified
8115        function with the given parameters.
8116
8117        :param operation: The `operation` parameter is a dictionary that contains information about the
8118        operation to be performed. It has the following keys:
8119        :type operation: dict
8120        :param operation_name: The `operation_name` parameter is a string that represents the name of
8121        the operation being performed. It is used for logging purposes, defaults to unknown
8122        :type operation_name: str (optional)
8123        """
8124
8125        operation_name = operation["name"]
8126        log.debug(f"process sql {operation_name}")
8127        function_name = operation["function_name"]
8128        function_params = operation["function_params"]
8129        getattr(self, function_name)(*function_params)

The calculation_process_function takes in an operation dictionary and performs the specified function with the given parameters.

Parameters

operation: The operation parameter is a dictionary that contains information about the operation to be performed. It has the following keys:
operation_name: The operation_name parameter is a string that represents the name of the operation being performed. It is used for logging purposes, defaults to unknown

def calculation_variant_id(self) -> None: View Source

8131    def calculation_variant_id(self) -> None:
8132        """
8133        The function `calculation_variant_id` adds a variant ID annotation to a VCF file header and
8134        updates the INFO field of a variants table with the variant ID.
8135        """
8136
8137        # variant_id annotation field
8138        variant_id_tag = self.get_variant_id_column()
8139        added_columns = [variant_id_tag]
8140
8141        # variant_id hgvs tags"
8142        vcf_infos_tags = {
8143            variant_id_tag: "howard variant ID annotation",
8144        }
8145
8146        # Variants table
8147        table_variants = self.get_table_variants()
8148
8149        # Header
8150        vcf_reader = self.get_header()
8151
8152        # Add variant_id to header
8153        vcf_reader.infos[variant_id_tag] = vcf.parser._Info(
8154            variant_id_tag,
8155            ".",
8156            "String",
8157            vcf_infos_tags.get(variant_id_tag, "howard variant ID annotation"),
8158            "howard calculation",
8159            "0",
8160            self.code_type_map.get("String"),
8161        )
8162
8163        # Update
8164        sql_update = f"""
8165            UPDATE {table_variants}
8166            SET "INFO" = 
8167                concat(
8168                    CASE
8169                        WHEN "INFO" IS NULL OR "INFO" IN ('','.')
8170                        THEN ''
8171                        ELSE concat("INFO", ';')
8172                    END,
8173                    '{variant_id_tag}=',
8174                    "{variant_id_tag}"
8175                )
8176        """
8177        self.conn.execute(sql_update)
8178
8179        # Remove added columns
8180        for added_column in added_columns:
8181            self.drop_column(column=added_column)

The function calculation_variant_id adds a variant ID annotation to a VCF file header and updates the INFO field of a variants table with the variant ID.

def calculation_extract_snpeff_hgvs( self, snpeff_hgvs: str = 'snpeff_hgvs', snpeff_field: str = 'ANN') -> None: View Source

8183    def calculation_extract_snpeff_hgvs(
8184        self,
8185        snpeff_hgvs: str = "snpeff_hgvs",
8186        snpeff_field: str = "ANN",
8187    ) -> None:
8188        """
8189        The function `calculation_extract_snpeff_hgvs` extracts HGVS nomenclatures from the SnpEff
8190        annotation field in a VCF file and adds them as a new column in the variants table.
8191
8192        :param snpeff_hgvs: The `snpeff_hgvs` parameter in the `calculation_extract_snpeff_hgvs`
8193        function is used to specify the name of the column that will store the HGVS nomenclatures
8194        extracted from the SnpEff annotation field in a VCF file. This parameter allows you, defaults to
8195        snpeff_hgvs
8196        :type snpeff_hgvs: str (optional)
8197        :param snpeff_field: The `snpeff_field` parameter in the `calculation_extract_snpeff_hgvs`
8198        function represents the field in the VCF file that contains SnpEff annotations. This field is
8199        used to extract HGVS nomenclatures from the SnpEff annotation field and add them as a, defaults
8200        to ANN
8201        :type snpeff_field: str (optional)
8202        """
8203
8204        # Snpeff hgvs tags
8205        vcf_infos_tags = {
8206            snpeff_hgvs: "HGVS nomenclatures from snpEff annotation",
8207        }
8208
8209        # Prefix
8210        prefix = self.get_explode_infos_prefix()
8211        if prefix:
8212            prefix = "INFO/"
8213
8214        # snpEff fields
8215        speff_ann_infos = prefix + snpeff_field
8216        speff_hgvs_infos = prefix + snpeff_hgvs
8217
8218        # Variants table
8219        table_variants = self.get_table_variants()
8220
8221        # Header
8222        vcf_reader = self.get_header()
8223
8224        # Add columns
8225        added_columns = []
8226
8227        # Explode HGVS field in column
8228        added_columns += self.explode_infos(fields=[snpeff_field])
8229
8230        if snpeff_field in vcf_reader.infos:
8231
8232            log.debug(vcf_reader.infos[snpeff_field])
8233
8234            # Extract ANN header
8235            ann_description = vcf_reader.infos[snpeff_field].desc
8236            pattern = r"'(.+?)'"
8237            match = re.search(pattern, ann_description)
8238            if match:
8239                ann_header_match = match.group(1).split(" | ")
8240                ann_header_desc = {}
8241                for i in range(len(ann_header_match)):
8242                    ann_header_info = "".join(
8243                        char for char in ann_header_match[i] if char.isalnum()
8244                    )
8245                    ann_header_desc[ann_header_info] = ann_header_match[i]
8246                if not ann_header_desc:
8247                    raise ValueError("Invalid header description format")
8248            else:
8249                raise ValueError("Invalid header description format")
8250
8251            # Create variant id
8252            variant_id_column = self.get_variant_id_column()
8253            added_columns += [variant_id_column]
8254
8255            # Create dataframe
8256            dataframe_snpeff_hgvs = self.get_query_to_df(
8257                f""" SELECT "{variant_id_column}", "{speff_ann_infos}" FROM {table_variants} """
8258            )
8259
8260            # Create main NOMEN column
8261            dataframe_snpeff_hgvs[speff_hgvs_infos] = dataframe_snpeff_hgvs[
8262                speff_ann_infos
8263            ].apply(
8264                lambda x: extract_snpeff_hgvs(
8265                    str(x), header=list(ann_header_desc.values())
8266                )
8267            )
8268
8269            # Add snpeff_hgvs to header
8270            vcf_reader.infos[snpeff_hgvs] = vcf.parser._Info(
8271                snpeff_hgvs,
8272                ".",
8273                "String",
8274                vcf_infos_tags.get(snpeff_hgvs, "snpEff hgvs annotations"),
8275                "howard calculation",
8276                "0",
8277                self.code_type_map.get("String"),
8278            )
8279
8280            # Update
8281            sql_update = f"""
8282                UPDATE variants
8283                SET "INFO" = 
8284                    concat(
8285                        CASE
8286                            WHEN "INFO" IS NULL OR "INFO" IN ('','.')
8287                            THEN ''
8288                            ELSE concat("INFO", ';')
8289                        END,
8290                        CASE 
8291                            WHEN dataframe_snpeff_hgvs."{speff_hgvs_infos}" NOT IN ('','.','NaN')
8292                            AND dataframe_snpeff_hgvs."{speff_hgvs_infos}" NOT NULL
8293                            THEN concat(
8294                                    '{snpeff_hgvs}=',
8295                                    dataframe_snpeff_hgvs."{speff_hgvs_infos}"
8296                                )
8297                            ELSE ''
8298                        END
8299                    )
8300                FROM dataframe_snpeff_hgvs
8301                WHERE {table_variants}."{variant_id_column}" = dataframe_snpeff_hgvs."{variant_id_column}"
8302
8303            """
8304            self.conn.execute(sql_update)
8305
8306            # Delete dataframe
8307            del dataframe_snpeff_hgvs
8308            gc.collect()
8309
8310        else:
8311
8312            log.warning(
8313                "No snpEff annotation. Please Anotate with snpEff before use this calculation option"
8314            )
8315
8316        # Remove added columns
8317        for added_column in added_columns:
8318            self.drop_column(column=added_column)

The function calculation_extract_snpeff_hgvs extracts HGVS nomenclatures from the SnpEff annotation field in a VCF file and adds them as a new column in the variants table.

Parameters

snpeff_hgvs: The snpeff_hgvs parameter in the calculation_extract_snpeff_hgvs function is used to specify the name of the column that will store the HGVS nomenclatures extracted from the SnpEff annotation field in a VCF file. This parameter allows you, defaults to snpeff_hgvs
snpeff_field: The snpeff_field parameter in the calculation_extract_snpeff_hgvs function represents the field in the VCF file that contains SnpEff annotations. This field is used to extract HGVS nomenclatures from the SnpEff annotation field and add them as a, defaults to ANN

def calculation_snpeff_ann_explode( self, uniquify: bool = True, output_format: str = 'fields', output_prefix: str = 'snpeff_', snpeff_field: str = 'ANN') -> None: View Source

8320    def calculation_snpeff_ann_explode(
8321        self,
8322        uniquify: bool = True,
8323        output_format: str = "fields",
8324        output_prefix: str = "snpeff_",
8325        snpeff_field: str = "ANN",
8326    ) -> None:
8327        """
8328        The `calculation_snpeff_ann_explode` function processes SnpEff annotations in a VCF file by
8329        exploding the HGVS field and updating variant information accordingly.
8330
8331        :param uniquify: The `uniquify` parameter in the `calculation_snpeff_ann_explode` method is a
8332        boolean flag that determines whether the output should be uniquified or not. When set to `True`,
8333        it indicates that the output should be unique, meaning that duplicate entries should be removed,
8334        defaults to True
8335        :type uniquify: bool (optional)
8336        :param output_format: The `output_format` parameter in the `calculation_snpeff_ann_explode`
8337        function specifies the format in which the output annotations will be generated. It has a
8338        default value of "fields". You can also set it to "JSON" to output the annotations in JSON
8339        format, defaults to fields
8340        :type output_format: str (optional)
8341        :param output_prefix: The `output_prefix` parameter in the `calculation_snpeff_ann_explode`
8342        method is used to specify the prefix that will be added to the output annotations generated
8343        during the calculation process. This prefix helps to differentiate the newly added annotations
8344        from existing ones in the output data. By default, the, defaults to ANN_
8345        :type output_prefix: str (optional)
8346        :param snpeff_field: The `snpeff_field` parameter in the `calculation_snpeff_ann_explode`
8347        function is used to specify the field in the VCF file that contains SnpEff annotations. This
8348        field will be processed to explode the HGVS annotations and update the variant information
8349        accordingly, defaults to ANN
8350        :type snpeff_field: str (optional)
8351        """
8352
8353        # SnpEff annotation field
8354        snpeff_hgvs = "snpeff_ann_explode"
8355
8356        # Snpeff hgvs tags
8357        vcf_infos_tags = {
8358            snpeff_hgvs: "Explode snpEff annotations",
8359        }
8360
8361        # Prefix
8362        prefix = self.get_explode_infos_prefix()
8363        if prefix:
8364            prefix = "INFO/"
8365
8366        # snpEff fields
8367        speff_ann_infos = prefix + snpeff_field
8368        speff_hgvs_infos = prefix + snpeff_hgvs
8369
8370        # Variants table
8371        table_variants = self.get_table_variants()
8372
8373        # Header
8374        vcf_reader = self.get_header()
8375
8376        # Add columns
8377        added_columns = []
8378
8379        # Explode HGVS field in column
8380        added_columns += self.explode_infos(fields=[snpeff_field])
8381        log.debug(f"snpeff_field={snpeff_field}")
8382        log.debug(f"added_columns={added_columns}")
8383
8384        if snpeff_field in vcf_reader.infos:
8385
8386            # Extract ANN header
8387            ann_description = vcf_reader.infos[snpeff_field].desc
8388            pattern = r"'(.+?)'"
8389            match = re.search(pattern, ann_description)
8390            if match:
8391                ann_header_match = match.group(1).split(" | ")
8392                ann_header = []
8393                ann_header_desc = {}
8394                for i in range(len(ann_header_match)):
8395                    ann_header_info = "".join(
8396                        char for char in ann_header_match[i] if char.isalnum()
8397                    )
8398                    ann_header.append(ann_header_info)
8399                    ann_header_desc[ann_header_info] = ann_header_match[i]
8400                if not ann_header_desc:
8401                    raise ValueError("Invalid header description format")
8402            else:
8403                raise ValueError("Invalid header description format")
8404
8405            # Create variant id
8406            variant_id_column = self.get_variant_id_column()
8407            added_columns += [variant_id_column]
8408
8409            # Create dataframe
8410            dataframe_snpeff_hgvs = self.get_query_to_df(
8411                f""" SELECT "{variant_id_column}", "{speff_ann_infos}" FROM {table_variants} """
8412            )
8413
8414            # Create snpEff columns
8415            dataframe_snpeff_hgvs[speff_hgvs_infos] = dataframe_snpeff_hgvs[
8416                speff_ann_infos
8417            ].apply(
8418                lambda x: explode_snpeff_ann(
8419                    str(x),
8420                    uniquify=uniquify,
8421                    output_format=output_format,
8422                    prefix=output_prefix,
8423                    header=list(ann_header_desc.values()),
8424                )
8425            )
8426
8427            # Header
8428            ann_annotations_prefix = ""
8429            if output_format.upper() in ["JSON"]:
8430                ann_annotations_prefix = f"{output_prefix}="
8431                vcf_reader.infos[output_prefix] = vcf.parser._Info(
8432                    output_prefix,
8433                    ".",
8434                    "String",
8435                    vcf_infos_tags.get(snpeff_hgvs, "snpEff annotations")
8436                    + " - JSON format",
8437                    "howard calculation",
8438                    "0",
8439                    self.code_type_map.get("String"),
8440                )
8441            else:
8442                for ann_annotation in ann_header:
8443                    ann_annotation_id = f"{output_prefix}{ann_annotation}"
8444                    vcf_reader.infos[ann_annotation_id] = vcf.parser._Info(
8445                        ann_annotation_id,
8446                        ".",
8447                        "String",
8448                        vcf_infos_tags.get(snpeff_hgvs, "snpEff annotations")
8449                        + f" - '{ann_header_desc[ann_annotation]}' annotation",
8450                        "howard calculation",
8451                        "0",
8452                        self.code_type_map.get("String"),
8453                    )
8454
8455            # Update
8456            sql_update = f"""
8457                UPDATE variants
8458                SET "INFO" = 
8459                    concat(
8460                        CASE
8461                            WHEN "INFO" IS NULL OR "INFO" IN ('','.')
8462                            THEN ''
8463                            ELSE concat("INFO", ';')
8464                        END,
8465                        CASE 
8466                            WHEN dataframe_snpeff_hgvs."{speff_hgvs_infos}" NOT IN ('','.','NaN')
8467                                AND dataframe_snpeff_hgvs."{speff_hgvs_infos}" NOT NULL
8468                            THEN concat(
8469                                '{ann_annotations_prefix}',
8470                                dataframe_snpeff_hgvs."{speff_hgvs_infos}"
8471                                )
8472                            ELSE ''
8473                        END
8474                    )
8475                FROM dataframe_snpeff_hgvs
8476                WHERE {table_variants}."{variant_id_column}" = dataframe_snpeff_hgvs."{variant_id_column}"
8477
8478            """
8479            self.conn.execute(sql_update)
8480
8481            # Delete dataframe
8482            del dataframe_snpeff_hgvs
8483            gc.collect()
8484
8485        else:
8486
8487            log.warning(
8488                "No snpEff annotation. Please Anotate with snpEff before use this calculation option"
8489            )
8490
8491        # Remove added columns
8492        for added_column in added_columns:
8493            self.drop_column(column=added_column)

The calculation_snpeff_ann_explode function processes SnpEff annotations in a VCF file by exploding the HGVS field and updating variant information accordingly.

Parameters

uniquify: The uniquify parameter in the calculation_snpeff_ann_explode method is a boolean flag that determines whether the output should be uniquified or not. When set to True, it indicates that the output should be unique, meaning that duplicate entries should be removed, defaults to True
output_format: The output_format parameter in the calculation_snpeff_ann_explode function specifies the format in which the output annotations will be generated. It has a default value of "fields". You can also set it to "JSON" to output the annotations in JSON format, defaults to fields
output_prefix: The output_prefix parameter in the calculation_snpeff_ann_explode method is used to specify the prefix that will be added to the output annotations generated during the calculation process. This prefix helps to differentiate the newly added annotations from existing ones in the output data. By default, the, defaults to ANN_
snpeff_field: The snpeff_field parameter in the calculation_snpeff_ann_explode function is used to specify the field in the VCF file that contains SnpEff annotations. This field will be processed to explode the HGVS annotations and update the variant information accordingly, defaults to ANN

def calculation_extract_nomen(self) -> None: View Source

8495    def calculation_extract_nomen(self) -> None:
8496        """
8497        This function extracts the HGVS nomenclature from the calculation/identification of NOMEN.
8498        """
8499
8500        # NOMEN field
8501        field_nomen_dict = "NOMEN_DICT"
8502
8503        # NOMEN structure
8504        nomen_dict = {
8505            "NOMEN": "NOMEN hgvs nomenclature considered as reference hgvs (official transcript, first otherwise)",
8506            "CNOMEN": "CNOMEN hgvs nomenclature at DNA level related to a transcript (TNOMEN)",
8507            "RNOMEN": "RNOMEN hgvs nomenclature at RNA level related to a transcript (TNOMEN)",
8508            "NNOMEN": "NNOMEN hgvs nomenclature for non-coding variant",
8509            "PNOMEN": "PNOMEN hgvs nomenclature at Protein level related to a transcript (TNOMEN)",
8510            "TVNOMEN": "TVNOMEN hgvs transcript with version (if any) used (e.g. for CNOMEN and PNOMEN)",
8511            "TNOMEN": "TNOMEN hgvs transcript used (e.g. for CNOMEN and PNOMEN)",
8512            "VNOMEN": "VNOMEN hgvs transcript version used (e.g. for CNOMEN and PNOMEN)",
8513            "ENOMEN": "ENOMEN hgvs exon nomenclature related to a transcript (TNOMEN)",
8514            "GNOMEN": "GNOMEN hgvs gene nomenclature related to a transcript (TNOMEN)",
8515        }
8516
8517        # Param
8518        param = self.get_param()
8519
8520        # Prefix
8521        prefix = self.get_explode_infos_prefix()
8522
8523        # Header
8524        vcf_reader = self.get_header()
8525
8526        # Get HGVS field
8527        hgvs_field = (
8528            param.get("calculation", {})
8529            .get("calculations", {})
8530            .get("NOMEN", {})
8531            .get("options", {})
8532            .get("hgvs_field", "hgvs")
8533        )
8534
8535        # Get transcripts
8536        transcripts_file = (
8537            param.get("calculation", {})
8538            .get("calculations", {})
8539            .get("NOMEN", {})
8540            .get("options", {})
8541            .get("transcripts", None)
8542        )
8543        transcripts_file = full_path(transcripts_file)
8544        transcripts = []
8545        if transcripts_file:
8546            if os.path.exists(transcripts_file):
8547                transcripts_dataframe = transcripts_file_to_df(transcripts_file)
8548                transcripts = transcripts_dataframe.iloc[:, 0].tolist()
8549            else:
8550                log.error(f"Transcript file '{transcripts_file}' does NOT exist")
8551                raise ValueError(f"Transcript file '{transcripts_file}' does NOT exist")
8552
8553        # Added columns
8554        added_columns = []
8555
8556        # Explode HGVS field in column
8557        added_columns += self.explode_infos(fields=[hgvs_field])
8558
8559        # extra infos
8560        extra_infos = self.get_extra_infos()
8561        extra_field = prefix + hgvs_field
8562
8563        if extra_field in extra_infos:
8564
8565            # Create dataframe
8566            dataframe_hgvs = self.get_query_to_df(
8567                f""" SELECT "#CHROM", "POS", "REF", "ALT", "{extra_field}" FROM variants """
8568            )
8569
8570            # Create main NOMEN column
8571            dataframe_hgvs[field_nomen_dict] = dataframe_hgvs[extra_field].apply(
8572                lambda x: find_nomen(str(x), transcripts=transcripts)
8573            )
8574
8575            # Explode NOMEN Structure and create SQL set for update
8576            sql_nomen_fields = []
8577            for nomen_field in nomen_dict:
8578
8579                # Explode each field into a column
8580                dataframe_hgvs[nomen_field] = dataframe_hgvs[field_nomen_dict].apply(
8581                    lambda x: dict(x).get(nomen_field, "")
8582                )
8583
8584                # Create VCF header field
8585                vcf_reader.infos[nomen_field] = vcf.parser._Info(
8586                    nomen_field,
8587                    ".",
8588                    "String",
8589                    nomen_dict.get(nomen_field, "howard calculation NOMEN"),
8590                    "howard calculation",
8591                    "0",
8592                    self.code_type_map.get("String"),
8593                )
8594                sql_nomen_fields.append(
8595                    f"""
8596                        CASE 
8597                            WHEN dataframe_hgvs."{nomen_field}" NOT NULL AND dataframe_hgvs."{nomen_field}" NOT IN ('')
8598                            THEN concat(
8599                                    ';{nomen_field}=',
8600                                    dataframe_hgvs."{nomen_field}"
8601                                )
8602                            ELSE ''
8603                        END
8604                    """
8605                )
8606
8607            # SQL set for update
8608            sql_nomen_fields_set = ", ".join(sql_nomen_fields)
8609
8610            # Update
8611            sql_update = f"""
8612                UPDATE variants
8613                SET "INFO" = 
8614                    concat(
8615                        CASE
8616                            WHEN "INFO" IS NULL
8617                            THEN ''
8618                            ELSE "INFO"
8619                        END,
8620                        {sql_nomen_fields_set}
8621                    )
8622                FROM dataframe_hgvs
8623                WHERE variants."#CHROM" = dataframe_hgvs."#CHROM"
8624                    AND variants."POS" = dataframe_hgvs."POS" 
8625                    AND variants."REF" = dataframe_hgvs."REF"
8626                    AND variants."ALT" = dataframe_hgvs."ALT"
8627            """
8628            self.conn.execute(sql_update)
8629
8630            # Delete dataframe
8631            del dataframe_hgvs
8632            gc.collect()
8633
8634        # Remove added columns
8635        for added_column in added_columns:
8636            self.drop_column(column=added_column)

This function extracts the HGVS nomenclature from the calculation/identification of NOMEN.

def calculation_find_by_pipeline(self, tag: str = 'findbypipeline') -> None: View Source

8638    def calculation_find_by_pipeline(self, tag: str = "findbypipeline") -> None:
8639        """
8640        The function `calculation_find_by_pipeline` performs a calculation to find the number of
8641        pipeline/sample for a variant and updates the variant information in a VCF file.
8642
8643        :param tag: The `tag` parameter is a string that represents the annotation field for the
8644        "findbypipeline" information in the VCF file. It is used to create the annotation field in the
8645        VCF header and to update the corresponding field in the variants table, defaults to
8646        findbypipeline
8647        :type tag: str (optional)
8648        """
8649
8650        # if FORMAT and samples
8651        if (
8652            "FORMAT" in self.get_header_columns_as_list()
8653            and self.get_header_sample_list()
8654        ):
8655
8656            # findbypipeline annotation field
8657            findbypipeline_tag = tag
8658
8659            # VCF infos tags
8660            vcf_infos_tags = {
8661                findbypipeline_tag: f"Number of pipeline/sample for a variant ({findbypipeline_tag})",
8662            }
8663
8664            # Prefix
8665            prefix = self.get_explode_infos_prefix()
8666
8667            # Field
8668            findbypipeline_infos = prefix + findbypipeline_tag
8669
8670            # Variants table
8671            table_variants = self.get_table_variants()
8672
8673            # Header
8674            vcf_reader = self.get_header()
8675
8676            # Create variant id
8677            variant_id_column = self.get_variant_id_column()
8678            added_columns = [variant_id_column]
8679
8680            # variant_id, FORMAT and samples
8681            samples_fields = f" {variant_id_column}, FORMAT , " + " , ".join(
8682                self.get_header_sample_list()
8683            )
8684
8685            # Create dataframe
8686            dataframe_findbypipeline = self.get_query_to_df(
8687                f""" SELECT {samples_fields} FROM {table_variants} """
8688            )
8689
8690            # Create findbypipeline column
8691            dataframe_findbypipeline[findbypipeline_infos] = (
8692                dataframe_findbypipeline.apply(
8693                    lambda row: findbypipeline(
8694                        row, samples=self.get_header_sample_list()
8695                    ),
8696                    axis=1,
8697                )
8698            )
8699
8700            # Add snpeff_hgvs to header
8701            vcf_reader.infos[findbypipeline_tag] = vcf.parser._Info(
8702                findbypipeline_tag,
8703                ".",
8704                "String",
8705                vcf_infos_tags.get(findbypipeline_tag, "Find in pipeline/sample"),
8706                "howard calculation",
8707                "0",
8708                self.code_type_map.get("String"),
8709            )
8710
8711            # Update
8712            sql_update = f"""
8713                UPDATE variants
8714                SET "INFO" = 
8715                    concat(
8716                        CASE
8717                            WHEN "INFO" IS NULL OR "INFO" IN ('','.')
8718                            THEN ''
8719                            ELSE concat("INFO", ';')
8720                        END,
8721                        CASE 
8722                            WHEN dataframe_findbypipeline."{findbypipeline_infos}" NOT IN ('','.')
8723                                AND dataframe_findbypipeline."{findbypipeline_infos}" NOT NULL
8724                            THEN concat(
8725                                    '{findbypipeline_tag}=',
8726                                    dataframe_findbypipeline."{findbypipeline_infos}"
8727                                )
8728                            ELSE ''
8729                        END
8730                    )
8731                FROM dataframe_findbypipeline
8732                WHERE variants."{variant_id_column}" = dataframe_findbypipeline."{variant_id_column}"
8733            """
8734            self.conn.execute(sql_update)
8735
8736            # Remove added columns
8737            for added_column in added_columns:
8738                self.drop_column(column=added_column)
8739
8740            # Delete dataframe
8741            del dataframe_findbypipeline
8742            gc.collect()

The function calculation_find_by_pipeline performs a calculation to find the number of pipeline/sample for a variant and updates the variant information in a VCF file.

Parameters

tag: The tag parameter is a string that represents the annotation field for the "findbypipeline" information in the VCF file. It is used to create the annotation field in the VCF header and to update the corresponding field in the variants table, defaults to findbypipeline

def calculation_genotype_concordance(self) -> None: View Source

8744    def calculation_genotype_concordance(self) -> None:
8745        """
8746        The function `calculation_genotype_concordance` calculates the genotype concordance for
8747        multi-caller VCF files and updates the variant information in the database.
8748        """
8749
8750        # if FORMAT and samples
8751        if (
8752            "FORMAT" in self.get_header_columns_as_list()
8753            and self.get_header_sample_list()
8754        ):
8755
8756            # genotypeconcordance annotation field
8757            genotypeconcordance_tag = "genotypeconcordance"
8758
8759            # VCF infos tags
8760            vcf_infos_tags = {
8761                genotypeconcordance_tag: "Concordance of genotype for multi caller VCF",
8762            }
8763
8764            # Prefix
8765            prefix = self.get_explode_infos_prefix()
8766
8767            # Field
8768            genotypeconcordance_infos = prefix + genotypeconcordance_tag
8769
8770            # Variants table
8771            table_variants = self.get_table_variants()
8772
8773            # Header
8774            vcf_reader = self.get_header()
8775
8776            # Create variant id
8777            variant_id_column = self.get_variant_id_column()
8778            added_columns = [variant_id_column]
8779
8780            # variant_id, FORMAT and samples
8781            samples_fields = f" {variant_id_column}, FORMAT , " + " , ".join(
8782                self.get_header_sample_list()
8783            )
8784
8785            # Create dataframe
8786            dataframe_genotypeconcordance = self.get_query_to_df(
8787                f""" SELECT {samples_fields} FROM {table_variants} """
8788            )
8789
8790            # Create genotypeconcordance column
8791            dataframe_genotypeconcordance[genotypeconcordance_infos] = (
8792                dataframe_genotypeconcordance.apply(
8793                    lambda row: genotypeconcordance(
8794                        row, samples=self.get_header_sample_list()
8795                    ),
8796                    axis=1,
8797                )
8798            )
8799
8800            # Add genotypeconcordance to header
8801            vcf_reader.infos[genotypeconcordance_tag] = vcf.parser._Info(
8802                genotypeconcordance_tag,
8803                ".",
8804                "String",
8805                vcf_infos_tags.get(genotypeconcordance_tag, "snpEff hgvs annotations"),
8806                "howard calculation",
8807                "0",
8808                self.code_type_map.get("String"),
8809            )
8810
8811            # Update
8812            sql_update = f"""
8813                UPDATE variants
8814                SET "INFO" = 
8815                    concat(
8816                        CASE
8817                            WHEN "INFO" IS NULL OR "INFO" IN ('','.')
8818                            THEN ''
8819                            ELSE concat("INFO", ';')
8820                        END,
8821                        CASE
8822                            WHEN dataframe_genotypeconcordance."{genotypeconcordance_infos}" NOT IN ('','.')
8823                                AND dataframe_genotypeconcordance."{genotypeconcordance_infos}" NOT NULL
8824                            THEN concat(
8825                                    '{genotypeconcordance_tag}=',
8826                                    dataframe_genotypeconcordance."{genotypeconcordance_infos}"
8827                                )
8828                            ELSE ''
8829                        END
8830                    )
8831                FROM dataframe_genotypeconcordance
8832                WHERE variants."{variant_id_column}" = dataframe_genotypeconcordance."{variant_id_column}"
8833            """
8834            self.conn.execute(sql_update)
8835
8836            # Remove added columns
8837            for added_column in added_columns:
8838                self.drop_column(column=added_column)
8839
8840            # Delete dataframe
8841            del dataframe_genotypeconcordance
8842            gc.collect()

The function calculation_genotype_concordance calculates the genotype concordance for multi-caller VCF files and updates the variant information in the database.

def calculation_barcode(self, tag: str = 'barcode') -> None: View Source

8844    def calculation_barcode(self, tag: str = "barcode") -> None:
8845        """
8846        The `calculation_barcode` function calculates barcode values for variants in a VCF file and
8847        updates the INFO field in the file with the calculated barcode values.
8848
8849        :param tag: The `tag` parameter in the `calculation_barcode` function is used to specify the tag
8850        name that will be used for the barcode calculation in the VCF file. If no tag name is provided,
8851        the default tag name is set to "barcode", defaults to barcode
8852        :type tag: str (optional)
8853        """
8854
8855        # if FORMAT and samples
8856        if (
8857            "FORMAT" in self.get_header_columns_as_list()
8858            and self.get_header_sample_list()
8859        ):
8860
8861            # barcode annotation field
8862            if not tag:
8863                tag = "barcode"
8864
8865            # VCF infos tags
8866            vcf_infos_tags = {
8867                tag: "barcode calculation (VaRank)",
8868            }
8869
8870            # Prefix
8871            prefix = self.get_explode_infos_prefix()
8872
8873            # Field
8874            barcode_infos = prefix + tag
8875
8876            # Variants table
8877            table_variants = self.get_table_variants()
8878
8879            # Header
8880            vcf_reader = self.get_header()
8881
8882            # Create variant id
8883            variant_id_column = self.get_variant_id_column()
8884            added_columns = [variant_id_column]
8885
8886            # variant_id, FORMAT and samples
8887            samples_fields = f" {variant_id_column}, FORMAT , " + " , ".join(
8888                self.get_header_sample_list()
8889            )
8890
8891            # Create dataframe
8892            dataframe_barcode = self.get_query_to_df(
8893                f""" SELECT {samples_fields} FROM {table_variants} """
8894            )
8895
8896            # Create barcode column
8897            dataframe_barcode[barcode_infos] = dataframe_barcode.apply(
8898                lambda row: barcode(row, samples=self.get_header_sample_list()), axis=1
8899            )
8900
8901            # Add barcode to header
8902            vcf_reader.infos[tag] = vcf.parser._Info(
8903                tag,
8904                ".",
8905                "String",
8906                vcf_infos_tags.get(tag, vcf_infos_tags.get(tag)),
8907                "howard calculation",
8908                "0",
8909                self.code_type_map.get("String"),
8910            )
8911
8912            # Update
8913            sql_update = f"""
8914                UPDATE {table_variants}
8915                SET "INFO" = 
8916                    concat(
8917                        CASE
8918                            WHEN "INFO" IS NULL OR "INFO" IN ('','.')
8919                            THEN ''
8920                            ELSE concat("INFO", ';')
8921                        END,
8922                        CASE
8923                            WHEN dataframe_barcode."{barcode_infos}" NOT IN ('','.')
8924                            AND dataframe_barcode."{barcode_infos}" NOT NULL
8925                            THEN concat(
8926                                    '{tag}=',
8927                                    dataframe_barcode."{barcode_infos}"
8928                                )
8929                            ELSE ''
8930                        END
8931                    )
8932                FROM dataframe_barcode
8933                WHERE {table_variants}."{variant_id_column}" = dataframe_barcode."{variant_id_column}"
8934            """
8935            self.conn.execute(sql_update)
8936
8937            # Remove added columns
8938            for added_column in added_columns:
8939                self.drop_column(column=added_column)
8940
8941            # Delete dataframe
8942            del dataframe_barcode
8943            gc.collect()

The calculation_barcode function calculates barcode values for variants in a VCF file and updates the INFO field in the file with the calculated barcode values.

Parameters

tag: The tag parameter in the calculation_barcode function is used to specify the tag name that will be used for the barcode calculation in the VCF file. If no tag name is provided, the default tag name is set to "barcode", defaults to barcode

def calculation_barcode_family(self, tag: str = 'BCF') -> None: View Source

8945    def calculation_barcode_family(self, tag: str = "BCF") -> None:
8946        """
8947        The `calculation_barcode_family` function calculates barcode values for variants in a VCF file
8948        and updates the INFO field in the file with the calculated barcode values.
8949
8950        :param tag: The `tag` parameter in the `calculation_barcode_family` function is used to specify
8951        the barcode tag that will be added to the VCF file during the calculation process. If no value
8952        is provided for the `tag` parameter, the default value used is "BCF", defaults to BCF
8953        :type tag: str (optional)
8954        """
8955
8956        # if FORMAT and samples
8957        if (
8958            "FORMAT" in self.get_header_columns_as_list()
8959            and self.get_header_sample_list()
8960        ):
8961
8962            # barcode annotation field
8963            if not tag:
8964                tag = "BCF"
8965
8966            # VCF infos tags
8967            vcf_infos_tags = {
8968                tag: "barcode family calculation",
8969                f"{tag}S": "barcode family samples",
8970            }
8971
8972            # Param
8973            param = self.get_param()
8974            log.debug(f"param={param}")
8975
8976            # Prefix
8977            prefix = self.get_explode_infos_prefix()
8978
8979            # PED param
8980            ped = (
8981                param.get("calculation", {})
8982                .get("calculations", {})
8983                .get("BARCODEFAMILY", {})
8984                .get("family_pedigree", None)
8985            )
8986            log.debug(f"ped={ped}")
8987
8988            # Load PED
8989            if ped:
8990
8991                # Pedigree is a file
8992                if isinstance(ped, str) and os.path.exists(full_path(ped)):
8993                    log.debug("Pedigree is file")
8994                    with open(full_path(ped)) as ped:
8995                        ped = json.load(ped)
8996
8997                # Pedigree is a string
8998                elif isinstance(ped, str):
8999                    log.debug("Pedigree is str")
9000                    try:
9001                        ped = json.loads(ped)
9002                        log.debug("Pedigree is json str")
9003                    except ValueError as e:
9004                        ped_samples = ped.split(",")
9005                        ped = {}
9006                        for ped_sample in ped_samples:
9007                            ped[ped_sample] = ped_sample
9008
9009                # Pedigree is a dict
9010                elif isinstance(ped, dict):
9011                    log.debug("Pedigree is dict")
9012
9013                # Pedigree is not well formatted
9014                else:
9015                    msg_error = "Pedigree not well formatted"
9016                    log.error(msg_error)
9017                    raise ValueError(msg_error)
9018
9019                # Construct list
9020                ped_samples = list(ped.values())
9021
9022            else:
9023                log.debug("Pedigree not defined. Take all samples")
9024                ped_samples = self.get_header_sample_list()
9025                ped = {}
9026                for ped_sample in ped_samples:
9027                    ped[ped_sample] = ped_sample
9028
9029            # Check pedigree
9030            if not ped or len(ped) == 0:
9031                msg_error = f"Error in pedigree: samples {ped_samples}"
9032                log.error(msg_error)
9033                raise ValueError(msg_error)
9034
9035            # Log
9036            log.info(
9037                "Calculation 'BARCODEFAMILY' - Samples: "
9038                + ", ".join([f"{member}='{ped[member]}'" for member in ped])
9039            )
9040            log.debug(f"ped_samples={ped_samples}")
9041
9042            # Field
9043            barcode_infos = prefix + tag
9044
9045            # Variants table
9046            table_variants = self.get_table_variants()
9047
9048            # Header
9049            vcf_reader = self.get_header()
9050
9051            # Create variant id
9052            variant_id_column = self.get_variant_id_column()
9053            added_columns = [variant_id_column]
9054
9055            # variant_id, FORMAT and samples
9056            samples_fields = f" {variant_id_column}, FORMAT , " + " , ".join(
9057                ped_samples
9058            )
9059
9060            # Create dataframe
9061            dataframe_barcode = self.get_query_to_df(
9062                f""" SELECT {samples_fields} FROM {table_variants} """
9063            )
9064
9065            # Create barcode column
9066            dataframe_barcode[barcode_infos] = dataframe_barcode.apply(
9067                lambda row: barcode(row, samples=ped_samples), axis=1
9068            )
9069
9070            # Add barcode family to header
9071            # Add vaf_normalization to header
9072            vcf_reader.formats[tag] = vcf.parser._Format(
9073                id=tag,
9074                num=".",
9075                type="String",
9076                desc=vcf_infos_tags.get(tag, "barcode family calculation"),
9077                type_code=self.code_type_map.get("String"),
9078            )
9079            vcf_reader.formats[f"{tag}S"] = vcf.parser._Format(
9080                id=f"{tag}S",
9081                num=".",
9082                type="String",
9083                desc=vcf_infos_tags.get(f"{tag}S", "barcode family samples"),
9084                type_code=self.code_type_map.get("String"),
9085            )
9086
9087            # Update
9088            # for sample in ped_samples:
9089            sql_update_set = []
9090            for sample in self.get_header_sample_list() + ["FORMAT"]:
9091                if sample in ped_samples:
9092                    value = f'dataframe_barcode."{barcode_infos}"'
9093                    value_samples = "'" + ",".join(ped_samples) + "'"
9094                elif sample == "FORMAT":
9095                    value = f"'{tag}'"
9096                    value_samples = f"'{tag}S'"
9097                else:
9098                    value = "'.'"
9099                    value_samples = "'.'"
9100                format_regex = r"[a-zA-Z0-9\s]"
9101                sql_update_set.append(
9102                    f"""
9103                        "{sample}" = 
9104                        concat(
9105                            CASE
9106                                WHEN {table_variants}."{sample}" = './.'
9107                                THEN concat('./.',regexp_replace(regexp_replace({table_variants}.FORMAT, '{format_regex}', '', 'g'), ':', ':.', 'g'))
9108                                ELSE {table_variants}."{sample}"
9109                            END,
9110                            ':',
9111                            {value},
9112                            ':',
9113                            {value_samples}
9114                        )
9115                    """
9116                )
9117
9118            sql_update_set_join = ", ".join(sql_update_set)
9119            sql_update = f"""
9120                UPDATE {table_variants}
9121                SET {sql_update_set_join}
9122                FROM dataframe_barcode
9123                WHERE {table_variants}."{variant_id_column}" = dataframe_barcode."{variant_id_column}"
9124            """
9125            self.conn.execute(sql_update)
9126
9127            # Remove added columns
9128            for added_column in added_columns:
9129                self.drop_column(column=added_column)
9130
9131            # Delete dataframe
9132            del dataframe_barcode
9133            gc.collect()

The calculation_barcode_family function calculates barcode values for variants in a VCF file and updates the INFO field in the file with the calculated barcode values.

Parameters

tag: The tag parameter in the calculation_barcode_family function is used to specify the barcode tag that will be added to the VCF file during the calculation process. If no value is provided for the tag parameter, the default value used is "BCF", defaults to BCF

def calculation_trio(self) -> None: View Source

9135    def calculation_trio(self) -> None:
9136        """
9137        The `calculation_trio` function performs trio calculations on a VCF file by adding trio
9138        information to the INFO field of each variant.
9139        """
9140
9141        # if FORMAT and samples
9142        if (
9143            "FORMAT" in self.get_header_columns_as_list()
9144            and self.get_header_sample_list()
9145        ):
9146
9147            # trio annotation field
9148            trio_tag = "trio"
9149
9150            # VCF infos tags
9151            vcf_infos_tags = {
9152                "trio": "trio calculation",
9153            }
9154
9155            # Param
9156            param = self.get_param()
9157
9158            # Prefix
9159            prefix = self.get_explode_infos_prefix()
9160
9161            # Trio param
9162            trio_ped = (
9163                param.get("calculation", {})
9164                .get("calculations", {})
9165                .get("TRIO", {})
9166                .get("trio_pedigree", None)
9167            )
9168
9169            # Load trio
9170            if trio_ped:
9171
9172                # Trio pedigree is a file
9173                if isinstance(trio_ped, str) and os.path.exists(full_path(trio_ped)):
9174                    log.debug("TRIO pedigree is file")
9175                    with open(full_path(trio_ped)) as trio_ped:
9176                        trio_ped = json.load(trio_ped)
9177
9178                # Trio pedigree is a string
9179                elif isinstance(trio_ped, str):
9180                    log.debug("TRIO pedigree is str")
9181                    try:
9182                        trio_ped = json.loads(trio_ped)
9183                        log.debug("TRIO pedigree is json str")
9184                    except ValueError as e:
9185                        trio_samples = trio_ped.split(",")
9186                        if len(trio_samples) == 3:
9187                            trio_ped = {
9188                                "father": trio_samples[0],
9189                                "mother": trio_samples[1],
9190                                "child": trio_samples[2],
9191                            }
9192                            log.debug("TRIO pedigree is list str")
9193                        else:
9194                            msg_error = "TRIO pedigree not well formatted"
9195                            log.error(msg_error)
9196                            raise ValueError(msg_error)
9197
9198                # Trio pedigree is a dict
9199                elif isinstance(trio_ped, dict):
9200                    log.debug("TRIO pedigree is dict")
9201
9202                # Trio pedigree is not well formatted
9203                else:
9204                    msg_error = "TRIO pedigree not well formatted"
9205                    log.error(msg_error)
9206                    raise ValueError(msg_error)
9207
9208                # Construct trio list
9209                trio_samples = [
9210                    trio_ped.get("father", ""),
9211                    trio_ped.get("mother", ""),
9212                    trio_ped.get("child", ""),
9213                ]
9214
9215            else:
9216                log.debug("TRIO pedigree not defined. Take the first 3 samples")
9217                samples_list = self.get_header_sample_list()
9218                if len(samples_list) >= 3:
9219                    trio_samples = self.get_header_sample_list()[0:3]
9220                    trio_ped = {
9221                        "father": trio_samples[0],
9222                        "mother": trio_samples[1],
9223                        "child": trio_samples[2],
9224                    }
9225                else:
9226                    msg_error = f"Error in TRIO pedigree: only {len(samples_list)} samples {samples_list}"
9227                    log.error(msg_error)
9228                    raise ValueError(msg_error)
9229
9230            # Check trio pedigree
9231            if not trio_ped or len(trio_ped) != 3:
9232                msg_error = f"Error in TRIO pedigree: {trio_ped}"
9233                log.error(msg_error)
9234                raise ValueError(msg_error)
9235
9236            # Log
9237            log.info(
9238                f"Calculation 'TRIO' - Samples: "
9239                + ", ".join([f"{member}='{trio_ped[member]}'" for member in trio_ped])
9240            )
9241
9242            # Field
9243            trio_infos = prefix + trio_tag
9244
9245            # Variants table
9246            table_variants = self.get_table_variants()
9247
9248            # Header
9249            vcf_reader = self.get_header()
9250
9251            # Create variant id
9252            variant_id_column = self.get_variant_id_column()
9253            added_columns = [variant_id_column]
9254
9255            # variant_id, FORMAT and samples
9256            samples_fields = f" {variant_id_column}, FORMAT , " + " , ".join(
9257                self.get_header_sample_list()
9258            )
9259
9260            # Create dataframe
9261            dataframe_trio = self.get_query_to_df(
9262                f""" SELECT {samples_fields} FROM {table_variants} """
9263            )
9264
9265            # Create trio column
9266            dataframe_trio[trio_infos] = dataframe_trio.apply(
9267                lambda row: trio(row, samples=trio_samples), axis=1
9268            )
9269
9270            # Add trio to header
9271            vcf_reader.infos[trio_tag] = vcf.parser._Info(
9272                trio_tag,
9273                ".",
9274                "String",
9275                vcf_infos_tags.get(trio_tag, "snpEff hgvs annotations"),
9276                "howard calculation",
9277                "0",
9278                self.code_type_map.get("String"),
9279            )
9280
9281            # Update
9282            sql_update = f"""
9283                UPDATE {table_variants}
9284                SET "INFO" = 
9285                    concat(
9286                        CASE
9287                            WHEN "INFO" IS NULL OR "INFO" IN ('','.')
9288                            THEN ''
9289                            ELSE concat("INFO", ';')
9290                        END,
9291                        CASE
9292                            WHEN dataframe_trio."{trio_infos}" NOT IN ('','.')
9293                             AND dataframe_trio."{trio_infos}" NOT NULL
9294                            THEN concat(
9295                                    '{trio_tag}=',
9296                                    dataframe_trio."{trio_infos}"
9297                                )
9298                            ELSE ''
9299                        END
9300                    )
9301                FROM dataframe_trio
9302                WHERE {table_variants}."{variant_id_column}" = dataframe_trio."{variant_id_column}"
9303            """
9304            self.conn.execute(sql_update)
9305
9306            # Remove added columns
9307            for added_column in added_columns:
9308                self.drop_column(column=added_column)
9309
9310            # Delete dataframe
9311            del dataframe_trio
9312            gc.collect()

The calculation_trio function performs trio calculations on a VCF file by adding trio information to the INFO field of each variant.

def calculation_vaf_normalization(self) -> None: View Source

9314    def calculation_vaf_normalization(self) -> None:
9315        """
9316        The `calculation_vaf_normalization` function calculates the VAF (Variant Allele Frequency)
9317        normalization for each sample in a VCF file and updates the FORMAT and INFO fields accordingly.
9318        :return: The function does not return anything.
9319        """
9320
9321        # if FORMAT and samples
9322        if (
9323            "FORMAT" in self.get_header_columns_as_list()
9324            and self.get_header_sample_list()
9325        ):
9326
9327            # vaf_normalization annotation field
9328            vaf_normalization_tag = "VAF"
9329
9330            # VCF infos tags
9331            vcf_infos_tags = {
9332                "VAF": "VAF Variant Frequency",
9333            }
9334
9335            # Prefix
9336            prefix = self.get_explode_infos_prefix()
9337
9338            # Variants table
9339            table_variants = self.get_table_variants()
9340
9341            # Header
9342            vcf_reader = self.get_header()
9343
9344            # Do not calculate if VAF already exists
9345            if "VAF" in vcf_reader.formats:
9346                log.debug("VAF already on genotypes")
9347                return
9348
9349            # Create variant id
9350            variant_id_column = self.get_variant_id_column()
9351            added_columns = [variant_id_column]
9352
9353            # variant_id, FORMAT and samples
9354            samples_fields = f" {variant_id_column}, FORMAT , " + " , ".join(
9355                f""" "{sample}" """ for sample in self.get_header_sample_list()
9356            )
9357
9358            # Create dataframe
9359            query = f""" SELECT {variant_id_column}, FORMAT, {samples_fields} FROM {table_variants} """
9360            log.debug(f"query={query}")
9361            dataframe_vaf_normalization = self.get_query_to_df(query=query)
9362
9363            vaf_normalization_set = []
9364
9365            # for each sample vaf_normalization
9366            for sample in self.get_header_sample_list():
9367                dataframe_vaf_normalization[sample] = dataframe_vaf_normalization.apply(
9368                    lambda row: vaf_normalization(row, sample=sample), axis=1
9369                )
9370                vaf_normalization_set.append(
9371                    f""" "{sample}" = dataframe_vaf_normalization."{sample}" """
9372                )
9373
9374            # Add VAF to FORMAT
9375            dataframe_vaf_normalization["FORMAT"] = dataframe_vaf_normalization[
9376                "FORMAT"
9377            ].apply(lambda x: str(x) + ":VAF")
9378            vaf_normalization_set.append(
9379                f""" "FORMAT" = dataframe_vaf_normalization."FORMAT" """
9380            )
9381
9382            # Add vaf_normalization to header
9383            vcf_reader.formats[vaf_normalization_tag] = vcf.parser._Format(
9384                id=vaf_normalization_tag,
9385                num="1",
9386                type="Float",
9387                desc=vcf_infos_tags.get(vaf_normalization_tag, "VAF Variant Frequency"),
9388                type_code=self.code_type_map.get("Float"),
9389            )
9390
9391            # Create fields to add in INFO
9392            sql_vaf_normalization_set = " , ".join(vaf_normalization_set)
9393
9394            # Update
9395            sql_update = f"""
9396                UPDATE {table_variants}
9397                SET {sql_vaf_normalization_set}
9398                FROM dataframe_vaf_normalization
9399                WHERE variants."{variant_id_column}" = dataframe_vaf_normalization."{variant_id_column}"
9400
9401            """
9402            self.conn.execute(sql_update)
9403
9404            # Remove added columns
9405            for added_column in added_columns:
9406                self.drop_column(column=added_column)
9407
9408            # Delete dataframe
9409            del dataframe_vaf_normalization
9410            gc.collect()

The calculation_vaf_normalization function calculates the VAF (Variant Allele Frequency) normalization for each sample in a VCF file and updates the FORMAT and INFO fields accordingly.

Returns

The function does not return anything.

def calculation_genotype_stats(self, info: str = 'VAF') -> None: View Source

9412    def calculation_genotype_stats(self, info: str = "VAF") -> None:
9413        """
9414        The `calculation_genotype_stats` function calculates genotype statistics for a given information
9415        field in a VCF file and updates the INFO column of the variants table with the calculated
9416        statistics.
9417
9418        :param info: The `info` parameter is a string that represents the type of information for which
9419        genotype statistics are calculated. It is used to generate various VCF info tags for the
9420        statistics, such as the number of occurrences, the list of values, the minimum value, the
9421        maximum value, the mean, the median, defaults to VAF
9422        :type info: str (optional)
9423        """
9424
9425        # if FORMAT and samples
9426        if (
9427            "FORMAT" in self.get_header_columns_as_list()
9428            and self.get_header_sample_list()
9429        ):
9430
9431            # vaf_stats annotation field
9432            vaf_stats_tag = info + "_stats"
9433
9434            # VCF infos tags
9435            vcf_infos_tags = {
9436                info + "_stats_nb": f"genotype {info} Statistics - number of {info}",
9437                info + "_stats_list": f"genotype {info} Statistics - list of {info}",
9438                info + "_stats_min": f"genotype {info} Statistics - min {info}",
9439                info + "_stats_max": f"genotype {info} Statistics - max {info}",
9440                info + "_stats_mean": f"genotype {info} Statistics - mean {info}",
9441                info + "_stats_mediane": f"genotype {info} Statistics - mediane {info}",
9442                info
9443                + "_stats_stdev": f"genotype {info} Statistics - standard deviation {info}",
9444            }
9445
9446            # Prefix
9447            prefix = self.get_explode_infos_prefix()
9448
9449            # Field
9450            vaf_stats_infos = prefix + vaf_stats_tag
9451
9452            # Variants table
9453            table_variants = self.get_table_variants()
9454
9455            # Header
9456            vcf_reader = self.get_header()
9457
9458            # Create variant id
9459            variant_id_column = self.get_variant_id_column()
9460            added_columns = [variant_id_column]
9461
9462            # variant_id, FORMAT and samples
9463            samples_fields = f" {variant_id_column}, FORMAT , " + " , ".join(
9464                self.get_header_sample_list()
9465            )
9466
9467            # Create dataframe
9468            dataframe_vaf_stats = self.get_query_to_df(
9469                f""" SELECT {samples_fields} FROM {table_variants} """
9470            )
9471
9472            # Create vaf_stats column
9473            dataframe_vaf_stats[vaf_stats_infos] = dataframe_vaf_stats.apply(
9474                lambda row: genotype_stats(
9475                    row, samples=self.get_header_sample_list(), info=info
9476                ),
9477                axis=1,
9478            )
9479
9480            # List of vcf tags
9481            sql_vaf_stats_fields = []
9482
9483            # Check all VAF stats infos
9484            for stat in vcf_infos_tags:
9485
9486                # Extract stats
9487                dataframe_vaf_stats[stat] = dataframe_vaf_stats[vaf_stats_infos].apply(
9488                    lambda x: dict(x).get(stat, "")
9489                )
9490
9491                # Add snpeff_hgvs to header
9492                vcf_reader.infos[stat] = vcf.parser._Info(
9493                    stat,
9494                    ".",
9495                    "String",
9496                    vcf_infos_tags.get(stat, "genotype statistics"),
9497                    "howard calculation",
9498                    "0",
9499                    self.code_type_map.get("String"),
9500                )
9501
9502                if len(sql_vaf_stats_fields):
9503                    sep = ";"
9504                else:
9505                    sep = ""
9506
9507                # Create fields to add in INFO
9508                sql_vaf_stats_fields.append(
9509                    f"""
9510                        CASE
9511                            WHEN dataframe_vaf_stats."{stat}" NOT NULL
9512                            THEN concat(
9513                                    '{sep}{stat}=',
9514                                    dataframe_vaf_stats."{stat}"
9515                                )
9516                            ELSE ''
9517                        END
9518                    """
9519                )
9520
9521            # SQL set for update
9522            sql_vaf_stats_fields_set = ",  ".join(sql_vaf_stats_fields)
9523
9524            # Update
9525            sql_update = f"""
9526                UPDATE {table_variants}
9527                SET "INFO" = 
9528                    concat(
9529                        CASE
9530                            WHEN "INFO" IS NULL OR "INFO" IN ('','.')
9531                            THEN ''
9532                            ELSE concat("INFO", ';')
9533                        END,
9534                        {sql_vaf_stats_fields_set}
9535                    )
9536                FROM dataframe_vaf_stats
9537                WHERE {table_variants}."{variant_id_column}" = dataframe_vaf_stats."{variant_id_column}"
9538
9539            """
9540            self.conn.execute(sql_update)
9541
9542            # Remove added columns
9543            for added_column in added_columns:
9544                self.drop_column(column=added_column)
9545
9546            # Delete dataframe
9547            del dataframe_vaf_stats
9548            gc.collect()

The calculation_genotype_stats function calculates genotype statistics for a given information field in a VCF file and updates the INFO column of the variants table with the calculated statistics.

Parameters

info: The info parameter is a string that represents the type of information for which genotype statistics are calculated. It is used to generate various VCF info tags for the statistics, such as the number of occurrences, the list of values, the minimum value, the maximum value, the mean, the median, defaults to VAF

def calculation_transcripts_annotation(self, info_json: str = None, info_format: str = None) -> None: View Source

9550    def calculation_transcripts_annotation(
9551        self, info_json: str = None, info_format: str = None
9552    ) -> None:
9553        """
9554        The `calculation_transcripts_annotation` function creates a transcripts table and adds an info
9555        field to it if transcripts are available.
9556
9557        :param info_json: The `info_json` parameter in the `calculation_transcripts_annotation` method
9558        is a string parameter that represents the information field to be used in the transcripts JSON.
9559        It is used to specify the JSON format for the transcripts information. If no value is provided
9560        when calling the method, it defaults to "
9561        :type info_json: str
9562        :param info_format: The `info_format` parameter in the `calculation_transcripts_annotation`
9563        method is a string parameter that specifies the format of the information field to be used in
9564        the transcripts JSON. It is used to define the format of the information field
9565        :type info_format: str
9566        """
9567
9568        # Create transcripts table
9569        transcripts_table = self.create_transcript_view()
9570
9571        # Add info field
9572        if transcripts_table:
9573            self.transcript_view_to_variants(
9574                transcripts_table=transcripts_table,
9575                transcripts_info_field_json=info_json,
9576                transcripts_info_field_format=info_format,
9577            )
9578        else:
9579            log.info("No Transcripts to process. Check param.json file configuration")

The calculation_transcripts_annotation function creates a transcripts table and adds an info field to it if transcripts are available.

Parameters

info_json: The info_json parameter in the calculation_transcripts_annotation method is a string parameter that represents the information field to be used in the transcripts JSON. It is used to specify the JSON format for the transcripts information. If no value is provided when calling the method, it defaults to "
info_format: The info_format parameter in the calculation_transcripts_annotation method is a string parameter that specifies the format of the information field to be used in the transcripts JSON. It is used to define the format of the information field

def calculation_transcripts_prioritization(self) -> None: View Source

9581    def calculation_transcripts_prioritization(self) -> None:
9582        """
9583        The function `calculation_transcripts_prioritization` creates a transcripts table and
9584        prioritizes transcripts based on certain criteria.
9585        """
9586
9587        # Create transcripts table
9588        transcripts_table = self.create_transcript_view()
9589
9590        # Add info field
9591        if transcripts_table:
9592            self.transcripts_prioritization(transcripts_table=transcripts_table)
9593        else:
9594            log.info("No Transcripts to process. Check param.json file configuration")

The function calculation_transcripts_prioritization creates a transcripts table and prioritizes transcripts based on certain criteria.

def transcripts_prioritization(self, transcripts_table: str = None, param: dict = {}) -> bool: View Source

9600    def transcripts_prioritization(
9601        self, transcripts_table: str = None, param: dict = {}
9602    ) -> bool:
9603        """
9604        The `transcripts_prioritization` function prioritizes transcripts based on certain parameters
9605        and updates the variants table with the prioritized information.
9606
9607        :param transcripts_table: The `transcripts_table` parameter is a string that specifies the name
9608        of the table containing transcripts data. If no value is provided, it defaults to "transcripts".
9609        This parameter is used to identify the table where the transcripts data is stored for the
9610        prioritization process
9611        :type transcripts_table: str
9612        :param param: The `param` parameter in the `transcripts_prioritization` method is a dictionary
9613        that contains various configuration settings for the prioritization process of transcripts. It
9614        is used to customize the behavior of the prioritization algorithm and includes settings such as
9615        the prefix for prioritization fields, default profiles, and other
9616        :type param: dict
9617        :return: The function `transcripts_prioritization` returns a boolean value `True` if the
9618        transcripts prioritization process is successfully completed, and `False` if there are any
9619        issues or if no profile is defined for transcripts prioritization.
9620        """
9621
9622        log.debug("Start transcripts prioritization...")
9623
9624        # Param
9625        if not param:
9626            param = self.get_param()
9627
9628        # Variants table
9629        table_variants = self.get_table_variants()
9630        log.debug(f"transcripts_table={transcripts_table}")
9631        # Transcripts table
9632        if transcripts_table is None:
9633            log.debug(f"transcripts_table={transcripts_table}")
9634            transcripts_table = self.create_transcript_view(
9635                transcripts_table="transcripts", param=param
9636            )
9637            log.debug(f"transcripts_table={transcripts_table}")
9638        if transcripts_table is None:
9639            msg_err = "No Transcripts table availalble"
9640            log.error(msg_err)
9641            raise ValueError(msg_err)
9642
9643        # Get transcripts columns
9644        columns_as_list_query = f"""
9645            DESCRIBE {transcripts_table}
9646        """
9647        columns_as_list = list(
9648            self.get_query_to_df(columns_as_list_query)["column_name"]
9649        )
9650
9651        # Create INFO if not exists
9652        if "INFO" not in columns_as_list:
9653            query_add_info = f"""
9654                ALTER TABLE {transcripts_table} ADD COLUMN INFO STRING DEFAULT '';
9655            """
9656            self.execute_query(query_add_info)
9657
9658        # Prioritization param and Force only PZ Score and Flag
9659        pz_param = param.get("transcripts", {}).get("prioritization", {})
9660        pz_fields_score = pz_param.get("pzprefix", "PTZ") + "Score"
9661        pz_fields_flag = pz_param.get("pzprefix", "PTZ") + "Flag"
9662        pz_fields_transcripts = pz_param.get("pzprefix", "PTZ") + "Transcript"
9663        pz_param["pzfields"] = [pz_fields_score, pz_fields_flag]
9664        pz_profile_default = (
9665            param.get("transcripts", {}).get("prioritization", {}).get("profiles", None)
9666        )
9667
9668        # Exit if no profile
9669        if pz_profile_default is None:
9670            log.warning("No profile defined for transcripts prioritization")
9671            return False
9672
9673        # Prioritization
9674        prioritization_result = self.prioritization(
9675            table=transcripts_table,
9676            pz_param=param.get("transcripts", {}).get("prioritization", {}),
9677        )
9678        if not prioritization_result:
9679            log.warning("Transcripts prioritization not processed")
9680            return False
9681
9682        # Explode PZ fields
9683        self.explode_infos(
9684            table=transcripts_table,
9685            fields=param.get("transcripts", {})
9686            .get("prioritization", {})
9687            .get("pzfields", []),
9688        )
9689
9690        # Export Transcripts prioritization infos to variants table
9691        query_update = f"""
9692            WITH RankedTranscripts AS (
9693                SELECT
9694                    "#CHROM", POS, REF, ALT, transcript, {pz_fields_score}, {pz_fields_flag},
9695                    ROW_NUMBER() OVER (
9696                        PARTITION BY "#CHROM", POS, REF, ALT
9697                        ORDER BY {pz_fields_flag} ASC, {pz_fields_score} DESC, transcript ASC
9698                    ) AS rn
9699                FROM
9700                    {transcripts_table}
9701            )
9702            UPDATE {table_variants}
9703                SET
9704                INFO = CONCAT(CASE
9705                            WHEN "INFO" IS NULL OR "INFO" IN ('','.')
9706                            THEN ''
9707                            ELSE concat("INFO", ';')
9708                        END,
9709                        concat('{pz_fields_transcripts}=', transcript, ';{pz_fields_score}=', {pz_fields_score}, ';{pz_fields_flag}=', {pz_fields_flag})
9710                        )
9711            FROM
9712                RankedTranscripts
9713            WHERE
9714                rn = 1
9715                AND variants."#CHROM" = RankedTranscripts."#CHROM"
9716                AND variants."POS" = RankedTranscripts."POS"
9717                AND variants."REF" = RankedTranscripts."REF"
9718                AND variants."ALT" = RankedTranscripts."ALT"
9719                
9720        """
9721        self.execute_query(query=query_update)
9722
9723        # Add PZ Transcript in header
9724        self.get_header().infos[pz_fields_transcripts] = vcf.parser._Info(
9725            pz_fields_transcripts,
9726            ".",
9727            "String",
9728            f"Transcript selected from transcripts prioritization process, profile {pz_profile_default}",
9729            "unknown",
9730            "unknown",
9731            code_type_map["String"],
9732        )
9733
9734        # Return
9735        return True

The transcripts_prioritization function prioritizes transcripts based on certain parameters and updates the variants table with the prioritized information.

Parameters

transcripts_table: The transcripts_table parameter is a string that specifies the name of the table containing transcripts data. If no value is provided, it defaults to "transcripts". This parameter is used to identify the table where the transcripts data is stored for the prioritization process
param: The param parameter in the transcripts_prioritization method is a dictionary that contains various configuration settings for the prioritization process of transcripts. It is used to customize the behavior of the prioritization algorithm and includes settings such as the prefix for prioritization fields, default profiles, and other

Returns

The function transcripts_prioritization returns a boolean value True if the transcripts prioritization process is successfully completed, and False if there are any issues or if no profile is defined for transcripts prioritization.

def create_transcript_view_from_columns_map( self, transcripts_table: str = 'transcripts', columns_maps: dict = {}, added_columns: list = [], temporary_tables: list = None, annotation_fields: list = None) -> tuple[list, list, list]: View Source

9737    def create_transcript_view_from_columns_map(
9738        self,
9739        transcripts_table: str = "transcripts",
9740        columns_maps: dict = {},
9741        added_columns: list = [],
9742        temporary_tables: list = None,
9743        annotation_fields: list = None,
9744    ) -> tuple[list, list, list]:
9745        """
9746        The `create_transcript_view_from_columns_map` function generates a temporary table view based on
9747        specified columns mapping for transcripts data.
9748
9749        :param transcripts_table: The `transcripts_table` parameter is a string that specifies the name of
9750        the table where the transcripts data is stored or will be stored in the database. This table
9751        typically contains information about transcripts such as Ensembl transcript IDs, gene names, scores,
9752        predictions, etc. It defaults to "transcripts, defaults to transcripts
9753        :type transcripts_table: str (optional)
9754        :param columns_maps: The `columns_maps` parameter is a dictionary that contains information about
9755        how to map columns from a transcripts table to create a view. Each entry in the `columns_maps` list
9756        represents a mapping configuration for a specific set of columns. It typically includes details such
9757        as the main transcript column and additional information columns
9758        :type columns_maps: dict
9759        :param added_columns: The `added_columns` parameter in the `create_transcript_view_from_columns_map`
9760        function is a list that stores the additional columns that will be added to the view being created
9761        based on the columns map provided. These columns are generated by exploding the transcript
9762        information columns along with the main transcript column
9763        :type added_columns: list
9764        :param temporary_tables: The `temporary_tables` parameter in the
9765        `create_transcript_view_from_columns_map` function is a list that stores the names of temporary
9766        tables created during the process of creating a transcript view from a columns map. These temporary
9767        tables are used to store intermediate results or transformations before the final view is generated
9768        :type temporary_tables: list
9769        :param annotation_fields: The `annotation_fields` parameter in the
9770        `create_transcript_view_from_columns_map` function is a list that stores the fields that are used
9771        for annotation in the query view creation process. These fields are extracted from the
9772        `transcripts_column` and `transcripts_infos_columns` specified in the `columns
9773        :type annotation_fields: list
9774        :return: The function `create_transcript_view_from_columns_map` returns a tuple containing three
9775        lists: `added_columns`, `temporary_tables`, and `annotation_fields`.
9776        """
9777
9778        log.debug("Start transcrpts view creation from columns map...")
9779
9780        # "from_columns_map": [
9781        #     {
9782        #         "transcripts_column": "Ensembl_transcriptid",
9783        #         "transcripts_infos_columns": [
9784        #             "genename",
9785        #             "Ensembl_geneid",
9786        #             "LIST_S2_score",
9787        #             "LIST_S2_pred",
9788        #         ],
9789        #     },
9790        #     {
9791        #         "transcripts_column": "Ensembl_transcriptid",
9792        #         "transcripts_infos_columns": [
9793        #             "genename",
9794        #             "VARITY_R_score",
9795        #             "Aloft_pred",
9796        #         ],
9797        #     },
9798        # ],
9799
9800        # Init
9801        if temporary_tables is None:
9802            temporary_tables = []
9803        if annotation_fields is None:
9804            annotation_fields = []
9805
9806        # Variants table
9807        table_variants = self.get_table_variants()
9808
9809        for columns_map in columns_maps:
9810
9811            # Transcript column
9812            transcripts_column = columns_map.get("transcripts_column", None)
9813
9814            # Transcripts infos columns
9815            transcripts_infos_columns = columns_map.get("transcripts_infos_columns", [])
9816
9817            if transcripts_column is not None:
9818
9819                # Explode
9820                added_columns += self.explode_infos(
9821                    fields=[transcripts_column] + transcripts_infos_columns
9822                )
9823
9824                # View clauses
9825                clause_select = []
9826                for field in [transcripts_column] + transcripts_infos_columns:
9827                    clause_select.append(
9828                        f""" regexp_split_to_table("{field}", ',') AS '{field}' """
9829                    )
9830                    if field not in [transcripts_column]:
9831                        annotation_fields.append(field)
9832
9833                # Querey View
9834                query = f""" 
9835                    SELECT
9836                        "#CHROM", POS, REF, ALT, INFO,
9837                        "{transcripts_column}" AS 'transcript',
9838                        {", ".join(clause_select)}
9839                    FROM (
9840                        SELECT 
9841                            "#CHROM", POS, REF, ALT, INFO,
9842                            {", ".join(clause_select)}
9843                        FROM {table_variants}
9844                        )
9845                    WHERE "{transcripts_column}" IS NOT NULL
9846                """
9847
9848                # Create temporary table
9849                temporary_table = transcripts_table + "".join(
9850                    random.choices(string.ascii_uppercase + string.digits, k=10)
9851                )
9852
9853                # Temporary_tables
9854                temporary_tables.append(temporary_table)
9855                query_view = f"""
9856                    CREATE TEMPORARY TABLE {temporary_table}
9857                    AS ({query})
9858                """
9859                self.execute_query(query=query_view)
9860
9861        return added_columns, temporary_tables, annotation_fields

The create_transcript_view_from_columns_map function generates a temporary table view based on specified columns mapping for transcripts data.

Parameters

transcripts_table: The transcripts_table parameter is a string that specifies the name of the table where the transcripts data is stored or will be stored in the database. This table typically contains information about transcripts such as Ensembl transcript IDs, gene names, scores, predictions, etc. It defaults to "transcripts, defaults to transcripts
columns_maps: The columns_maps parameter is a dictionary that contains information about how to map columns from a transcripts table to create a view. Each entry in the columns_maps list represents a mapping configuration for a specific set of columns. It typically includes details such as the main transcript column and additional information columns
added_columns: The added_columns parameter in the create_transcript_view_from_columns_map function is a list that stores the additional columns that will be added to the view being created based on the columns map provided. These columns are generated by exploding the transcript information columns along with the main transcript column
temporary_tables: The temporary_tables parameter in the create_transcript_view_from_columns_map function is a list that stores the names of temporary tables created during the process of creating a transcript view from a columns map. These temporary tables are used to store intermediate results or transformations before the final view is generated
annotation_fields: The annotation_fields parameter in the create_transcript_view_from_columns_map function is a list that stores the fields that are used for annotation in the query view creation process. These fields are extracted from the transcripts_column and transcripts_infos_columns specified in the `columns

Returns

The function create_transcript_view_from_columns_map returns a tuple containing three lists: added_columns, temporary_tables, and annotation_fields.

def create_transcript_view_from_column_format( self, transcripts_table: str = 'transcripts', column_formats: dict = {}, temporary_tables: list = None, annotation_fields: list = None) -> tuple[list, list, list]: View Source

9863    def create_transcript_view_from_column_format(
9864        self,
9865        transcripts_table: str = "transcripts",
9866        column_formats: dict = {},
9867        temporary_tables: list = None,
9868        annotation_fields: list = None,
9869    ) -> tuple[list, list, list]:
9870        """
9871        The `create_transcript_view_from_column_format` function generates a transcript view based on
9872        specified column formats, adds additional columns and annotation fields, and returns the list of
9873        temporary tables and annotation fields.
9874
9875        :param transcripts_table: The `transcripts_table` parameter is a string that specifies the name of
9876        the table containing the transcripts data. This table will be used as the base table for creating
9877        the transcript view. The default value for this parameter is "transcripts", but you can provide a
9878        different table name if needed, defaults to transcripts
9879        :type transcripts_table: str (optional)
9880        :param column_formats: The `column_formats` parameter is a dictionary that contains information
9881        about the columns to be used for creating the transcript view. Each entry in the dictionary
9882        specifies the mapping between a transcripts column and a transcripts infos column. For example, in
9883        the provided code snippet:
9884        :type column_formats: dict
9885        :param temporary_tables: The `temporary_tables` parameter in the
9886        `create_transcript_view_from_column_format` function is a list that stores the names of temporary
9887        views created during the process of creating a transcript view from a column format. These temporary
9888        views are used to manipulate and extract data before generating the final transcript view. It
9889        :type temporary_tables: list
9890        :param annotation_fields: The `annotation_fields` parameter in the
9891        `create_transcript_view_from_column_format` function is a list that stores the annotation fields
9892        that are extracted from the temporary views created during the process. These annotation fields are
9893        obtained by querying the temporary views and extracting the column names excluding specific columns
9894        like `#CH
9895        :type annotation_fields: list
9896        :return: The `create_transcript_view_from_column_format` function returns two lists:
9897        `temporary_tables` and `annotation_fields`.
9898        """
9899
9900        log.debug("Start transcrpts view creation from column format...")
9901
9902        #  "from_column_format": [
9903        #     {
9904        #         "transcripts_column": "ANN",
9905        #         "transcripts_infos_column": "Feature_ID",
9906        #     }
9907        # ],
9908
9909        # Init
9910        if temporary_tables is None:
9911            temporary_tables = []
9912        if annotation_fields is None:
9913            annotation_fields = []
9914
9915        for column_format in column_formats:
9916
9917            # annotation field and transcript annotation field
9918            annotation_field = column_format.get("transcripts_column", "ANN")
9919            transcript_annotation = column_format.get(
9920                "transcripts_infos_column", "Feature_ID"
9921            )
9922
9923            # Temporary View name
9924            temporary_view_name = transcripts_table + "".join(
9925                random.choices(string.ascii_uppercase + string.digits, k=10)
9926            )
9927
9928            # Create temporary view name
9929            temporary_view_name = self.annotation_format_to_table(
9930                uniquify=True,
9931                annotation_field=annotation_field,
9932                view_name=temporary_view_name,
9933                annotation_id=transcript_annotation,
9934            )
9935
9936            # Annotation fields
9937            if temporary_view_name:
9938                query_annotation_fields = f"""
9939                    SELECT *
9940                    FROM (
9941                        DESCRIBE SELECT *
9942                        FROM {temporary_view_name}
9943                        )
9944                        WHERE column_name not in ('#CHROM', 'POS', 'REF', 'ALT')
9945                """
9946                df_annotation_fields = self.get_query_to_df(
9947                    query=query_annotation_fields
9948                )
9949
9950                # Add temporary view and annotation fields
9951                temporary_tables.append(temporary_view_name)
9952                annotation_fields += list(set(df_annotation_fields["column_name"]))
9953
9954        return temporary_tables, annotation_fields

The create_transcript_view_from_column_format function generates a transcript view based on specified column formats, adds additional columns and annotation fields, and returns the list of temporary tables and annotation fields.

Parameters

transcripts_table: The transcripts_table parameter is a string that specifies the name of the table containing the transcripts data. This table will be used as the base table for creating the transcript view. The default value for this parameter is "transcripts", but you can provide a different table name if needed, defaults to transcripts
column_formats: The column_formats parameter is a dictionary that contains information about the columns to be used for creating the transcript view. Each entry in the dictionary specifies the mapping between a transcripts column and a transcripts infos column. For example, in the provided code snippet:
temporary_tables: The temporary_tables parameter in the create_transcript_view_from_column_format function is a list that stores the names of temporary views created during the process of creating a transcript view from a column format. These temporary views are used to manipulate and extract data before generating the final transcript view. It
annotation_fields: The annotation_fields parameter in the create_transcript_view_from_column_format function is a list that stores the annotation fields that are extracted from the temporary views created during the process. These annotation fields are obtained by querying the temporary views and extracting the column names excluding specific columns like `#CH

Returns

The create_transcript_view_from_column_format function returns two lists: temporary_tables and annotation_fields.

def create_transcript_view( self, transcripts_table: str = None, transcripts_table_drop: bool = True, param: dict = {}) -> str: View Source

 9956    def create_transcript_view(
 9957        self,
 9958        transcripts_table: str = None,
 9959        transcripts_table_drop: bool = True,
 9960        param: dict = {},
 9961    ) -> str:
 9962        """
 9963        The `create_transcript_view` function generates a transcript view by processing data from a
 9964        specified table based on provided parameters and structural information.
 9965
 9966        :param transcripts_table: The `transcripts_table` parameter in the `create_transcript_view` function
 9967        is used to specify the name of the table that will store the final transcript view data. If a table
 9968        name is not provided, the function will create a new table to store the transcript view data, and by
 9969        default,, defaults to transcripts
 9970        :type transcripts_table: str (optional)
 9971        :param transcripts_table_drop: The `transcripts_table_drop` parameter in the
 9972        `create_transcript_view` function is a boolean parameter that determines whether to drop the
 9973        existing transcripts table before creating a new one. If `transcripts_table_drop` is set to `True`,
 9974        the function will drop the existing transcripts table if it exists, defaults to True
 9975        :type transcripts_table_drop: bool (optional)
 9976        :param param: The `param` parameter in the `create_transcript_view` function is a dictionary that
 9977        contains information needed to create a transcript view. It includes details such as the structure
 9978        of the transcripts, columns mapping, column formats, and other necessary information for generating
 9979        the view. This parameter allows for flexibility and customization
 9980        :type param: dict
 9981        :return: The `create_transcript_view` function returns the name of the transcripts table that was
 9982        created or modified during the execution of the function.
 9983        """
 9984
 9985        log.debug("Start transcripts view creation...")
 9986
 9987        # Default
 9988        transcripts_table_default = "transcripts"
 9989
 9990        # Param
 9991        if not param:
 9992            param = self.get_param()
 9993
 9994        # Struct
 9995        struct = param.get("transcripts", {}).get("struct", None)
 9996
 9997        if struct:
 9998
 9999            # Transcripts table
10000            if transcripts_table is None:
10001                transcripts_table = param.get("transcripts", {}).get(
10002                    "table", transcripts_table_default
10003                )
10004
10005            # added_columns
10006            added_columns = []
10007
10008            # Temporary tables
10009            temporary_tables = []
10010
10011            # Annotation fields
10012            annotation_fields = []
10013
10014            # from columns map
10015            columns_maps = struct.get("from_columns_map", [])
10016            added_columns_tmp, temporary_tables_tmp, annotation_fields_tmp = (
10017                self.create_transcript_view_from_columns_map(
10018                    transcripts_table=transcripts_table,
10019                    columns_maps=columns_maps,
10020                    added_columns=added_columns,
10021                    temporary_tables=temporary_tables,
10022                    annotation_fields=annotation_fields,
10023                )
10024            )
10025            added_columns += added_columns_tmp
10026            temporary_tables += temporary_tables_tmp
10027            annotation_fields += annotation_fields_tmp
10028
10029            # from column format
10030            column_formats = struct.get("from_column_format", [])
10031            temporary_tables_tmp, annotation_fields_tmp = (
10032                self.create_transcript_view_from_column_format(
10033                    transcripts_table=transcripts_table,
10034                    column_formats=column_formats,
10035                    temporary_tables=temporary_tables,
10036                    annotation_fields=annotation_fields,
10037                )
10038            )
10039            temporary_tables += temporary_tables_tmp
10040            annotation_fields += annotation_fields_tmp
10041
10042            # Merge temporary tables query
10043            query_merge = ""
10044            for temporary_table in temporary_tables:
10045
10046                # First temporary table
10047                if not query_merge:
10048                    query_merge = f"""
10049                        SELECT * FROM {temporary_table}
10050                    """
10051                # other temporary table (using UNION)
10052                else:
10053                    query_merge += f"""
10054                        UNION BY NAME SELECT * FROM {temporary_table}
10055                    """
10056
10057            # Merge on transcript
10058            query_merge_on_transcripts_annotation_fields = []
10059            # Aggregate all annotations fields
10060            for annotation_field in set(annotation_fields):
10061                query_merge_on_transcripts_annotation_fields.append(
10062                    f""" list_aggregate(list_distinct(array_agg({annotation_field})), 'string_agg', ',') AS {annotation_field} """
10063                )
10064            # Query for transcripts view
10065            query_merge_on_transcripts = f"""
10066                SELECT "#CHROM", POS, REF, ALT, INFO, transcript, {", ".join(query_merge_on_transcripts_annotation_fields)}
10067                FROM ({query_merge})
10068                GROUP BY "#CHROM", POS, REF, ALT, INFO, transcript
10069            """
10070
10071            # Drop transcript view is necessary
10072            if transcripts_table_drop:
10073                query_drop = f"""
10074                    DROP TABLE IF EXISTS {transcripts_table};
10075                """
10076                self.execute_query(query=query_drop)
10077
10078            # Merge and create transcript view
10079            query_create_view = f"""
10080                CREATE TABLE IF NOT EXISTS {transcripts_table}
10081                AS {query_merge_on_transcripts}
10082            """
10083            self.execute_query(query=query_create_view)
10084
10085            # Remove added columns
10086            for added_column in added_columns:
10087                self.drop_column(column=added_column)
10088
10089        else:
10090
10091            transcripts_table = None
10092
10093        return transcripts_table

The create_transcript_view function generates a transcript view by processing data from a specified table based on provided parameters and structural information.

Parameters

transcripts_table: The transcripts_table parameter in the create_transcript_view function is used to specify the name of the table that will store the final transcript view data. If a table name is not provided, the function will create a new table to store the transcript view data, and by default,, defaults to transcripts
transcripts_table_drop: The transcripts_table_drop parameter in the create_transcript_view function is a boolean parameter that determines whether to drop the existing transcripts table before creating a new one. If transcripts_table_drop is set to True, the function will drop the existing transcripts table if it exists, defaults to True
param: The param parameter in the create_transcript_view function is a dictionary that contains information needed to create a transcript view. It includes details such as the structure of the transcripts, columns mapping, column formats, and other necessary information for generating the view. This parameter allows for flexibility and customization

Returns

The create_transcript_view function returns the name of the transcripts table that was created or modified during the execution of the function.

def annotation_format_to_table( self, uniquify: bool = True, annotation_field: str = 'ANN', annotation_id: str = 'Feature_ID', view_name: str = 'transcripts') -> str: View Source

10095    def annotation_format_to_table(
10096        self,
10097        uniquify: bool = True,
10098        annotation_field: str = "ANN",
10099        annotation_id: str = "Feature_ID",
10100        view_name: str = "transcripts",
10101    ) -> str:
10102        """
10103        The function `annotation_format_to_table` converts annotation data from a VCF file into a structured
10104        table format.
10105
10106        :param uniquify: The `uniquify` parameter is a boolean flag that determines whether to ensure unique
10107        values in the output or not. If set to `True`, the function will make sure that the output values
10108        are unique, defaults to True
10109        :type uniquify: bool (optional)
10110        :param annotation_field: The `annotation_field` parameter refers to the field in the VCF file that
10111        contains the annotation information for each variant. This field is used to extract the annotation
10112        details for further processing in the function, defaults to ANN
10113        :type annotation_field: str (optional)
10114        :param annotation_id: The `annotation_id` parameter in the `annotation_format_to_table` method is
10115        used to specify the identifier for the annotation feature. This identifier will be used as a column
10116        name in the resulting table or view that is created based on the annotation data. It helps in
10117        uniquely identifying each annotation entry in the, defaults to Feature_ID
10118        :type annotation_id: str (optional)
10119        :param view_name: The `view_name` parameter in the `annotation_format_to_table` method is used to
10120        specify the name of the temporary table that will be created to store the transformed annotation
10121        data. This table will hold the extracted information from the annotation field in a structured
10122        format for further processing or analysis, defaults to transcripts
10123        :type view_name: str (optional)
10124        :return: The function `annotation_format_to_table` is returning the name of the view created, which
10125        is stored in the variable `view_name`.
10126        """
10127
10128        # Annotation field
10129        annotation_format = "annotation_explode"
10130
10131        # Transcript annotation
10132        annotation_id = "".join(char for char in annotation_id if char.isalnum())
10133
10134        # Prefix
10135        prefix = self.get_explode_infos_prefix()
10136        if prefix:
10137            prefix = "INFO/"
10138
10139        # Annotation fields
10140        annotation_infos = prefix + annotation_field
10141        annotation_format_infos = prefix + annotation_format
10142
10143        # Variants table
10144        table_variants = self.get_table_variants()
10145
10146        # Header
10147        vcf_reader = self.get_header()
10148
10149        # Add columns
10150        added_columns = []
10151
10152        # Explode HGVS field in column
10153        added_columns += self.explode_infos(fields=[annotation_field])
10154
10155        if annotation_field in vcf_reader.infos:
10156
10157            # Extract ANN header
10158            ann_description = vcf_reader.infos[annotation_field].desc
10159            pattern = r"'(.+?)'"
10160            match = re.search(pattern, ann_description)
10161            if match:
10162                ann_header_match = match.group(1).split(" | ")
10163                ann_header = []
10164                ann_header_desc = {}
10165                for i in range(len(ann_header_match)):
10166                    ann_header_info = "".join(
10167                        char for char in ann_header_match[i] if char.isalnum()
10168                    )
10169                    ann_header.append(ann_header_info)
10170                    ann_header_desc[ann_header_info] = ann_header_match[i]
10171                if not ann_header_desc:
10172                    raise ValueError("Invalid header description format")
10173            else:
10174                raise ValueError("Invalid header description format")
10175
10176            # Create variant id
10177            variant_id_column = self.get_variant_id_column()
10178            added_columns += [variant_id_column]
10179
10180            # Create dataframe
10181            dataframe_annotation_format = self.get_query_to_df(
10182                f""" SELECT "#CHROM", POS, REF, ALT, INFO, "{variant_id_column}", "{annotation_infos}" FROM {table_variants} """
10183            )
10184
10185            # Create annotation columns
10186            dataframe_annotation_format[
10187                annotation_format_infos
10188            ] = dataframe_annotation_format[annotation_infos].apply(
10189                lambda x: explode_annotation_format(
10190                    annotation=str(x),
10191                    uniquify=uniquify,
10192                    output_format="JSON",
10193                    prefix="",
10194                    header=list(ann_header_desc.values()),
10195                )
10196            )
10197
10198            # Find keys
10199            query_json = f"""SELECT distinct(unnest(json_keys({annotation_format}, '$.0'))) AS 'key' FROM dataframe_annotation_format;"""
10200            df_keys = self.get_query_to_df(query=query_json)
10201
10202            # Check keys
10203            query_json_key = []
10204            for _, row in df_keys.iterrows():
10205
10206                # Key
10207                key = row.iloc[0]
10208
10209                # key_clean
10210                key_clean = "".join(char for char in key if char.isalnum())
10211
10212                # Type
10213                query_json_type = f"""SELECT unnest(json_extract_string({annotation_format}, '$.*."{key}"')) AS '{key_clean}' FROM dataframe_annotation_format WHERE trim('{key}') NOT IN ('');"""
10214
10215                # Get DataFrame from query
10216                df_json_type = self.get_query_to_df(query=query_json_type)
10217
10218                # Fill missing values with empty strings and then replace empty strings or None with NaN and drop rows with NaN
10219                with pd.option_context("future.no_silent_downcasting", True):
10220                    df_json_type.fillna(value="", inplace=True)
10221                    replace_dict = {None: np.nan, "": np.nan}
10222                    df_json_type.replace(replace_dict, inplace=True)
10223                    df_json_type.dropna(inplace=True)
10224
10225                # Detect column type
10226                column_type = detect_column_type(df_json_type[key_clean])
10227
10228                # Append
10229                query_json_key.append(
10230                    f"""NULLIF(unnest(json_extract_string({annotation_format}, '$.*."{key}"')), '')::{column_type}  AS '{prefix}{key_clean}' """
10231                )
10232
10233            # Create view
10234            query_view = f"""
10235                CREATE TEMPORARY TABLE {view_name}
10236                AS (
10237                    SELECT *, {annotation_id} AS 'transcript'
10238                    FROM (
10239                        SELECT "#CHROM", POS, REF, ALT, INFO, {",".join(query_json_key)}
10240                        FROM dataframe_annotation_format
10241                        )
10242                    );
10243            """
10244            self.execute_query(query=query_view)
10245
10246        else:
10247
10248            # Return None
10249            view_name = None
10250
10251        # Remove added columns
10252        for added_column in added_columns:
10253            self.drop_column(column=added_column)
10254
10255        return view_name

The function annotation_format_to_table converts annotation data from a VCF file into a structured table format.

Parameters

uniquify: The uniquify parameter is a boolean flag that determines whether to ensure unique values in the output or not. If set to True, the function will make sure that the output values are unique, defaults to True
annotation_field: The annotation_field parameter refers to the field in the VCF file that contains the annotation information for each variant. This field is used to extract the annotation details for further processing in the function, defaults to ANN
annotation_id: The annotation_id parameter in the annotation_format_to_table method is used to specify the identifier for the annotation feature. This identifier will be used as a column name in the resulting table or view that is created based on the annotation data. It helps in uniquely identifying each annotation entry in the, defaults to Feature_ID
view_name: The view_name parameter in the annotation_format_to_table method is used to specify the name of the temporary table that will be created to store the transformed annotation data. This table will hold the extracted information from the annotation field in a structured format for further processing or analysis, defaults to transcripts

Returns

The function annotation_format_to_table is returning the name of the view created, which is stored in the variable view_name.

def transcript_view_to_variants( self, transcripts_table: str = None, transcripts_column_id: str = None, transcripts_info_json: str = None, transcripts_info_field_json: str = None, transcripts_info_format: str = None, transcripts_info_field_format: str = None, param: dict = {}) -> bool: View Source

10257    def transcript_view_to_variants(
10258        self,
10259        transcripts_table: str = None,
10260        transcripts_column_id: str = None,
10261        transcripts_info_json: str = None,
10262        transcripts_info_field_json: str = None,
10263        transcripts_info_format: str = None,
10264        transcripts_info_field_format: str = None,
10265        param: dict = {},
10266    ) -> bool:
10267        """
10268        The `transcript_view_to_variants` function updates a variants table with information from
10269        transcripts in JSON format.
10270
10271        :param transcripts_table: The `transcripts_table` parameter is used to specify the name of the
10272        table containing the transcripts data. If this parameter is not provided, the function will
10273        attempt to retrieve it from the `param` dictionary or use a default value of "transcripts"
10274        :type transcripts_table: str
10275        :param transcripts_column_id: The `transcripts_column_id` parameter is used to specify the
10276        column in the `transcripts_table` that contains the unique identifier for each transcript. This
10277        identifier is used to match transcripts with variants in the database
10278        :type transcripts_column_id: str
10279        :param transcripts_info_json: The `transcripts_info_json` parameter is used to specify the name
10280        of the column in the variants table where the transcripts information will be stored in JSON
10281        format. This parameter allows you to define the column in the variants table that will hold the
10282        JSON-formatted information about transcripts
10283        :type transcripts_info_json: str
10284        :param transcripts_info_field_json: The `transcripts_info_field_json` parameter is used to
10285        specify the field in the VCF header that will contain information about transcripts in JSON
10286        format. This field will be added to the VCF header as an INFO field with the specified name
10287        :type transcripts_info_field_json: str
10288        :param transcripts_info_format: The `transcripts_info_format` parameter is used to specify the
10289        format of the information about transcripts that will be stored in the variants table. This
10290        format can be used to define how the transcript information will be structured or displayed
10291        within the variants table
10292        :type transcripts_info_format: str
10293        :param transcripts_info_field_format: The `transcripts_info_field_format` parameter is used to
10294        specify the field in the VCF header that will contain information about transcripts in a
10295        specific format. This field will be added to the VCF header as an INFO field with the specified
10296        name
10297        :type transcripts_info_field_format: str
10298        :param param: The `param` parameter in the `transcript_view_to_variants` method is a dictionary
10299        that contains various configuration settings related to transcripts. It is used to provide
10300        default values for certain parameters if they are not explicitly provided when calling the
10301        method. The `param` dictionary can be passed as an argument
10302        :type param: dict
10303        :return: The function `transcript_view_to_variants` returns a boolean value. It returns `True`
10304        if the operation is successful and `False` if certain conditions are not met.
10305        """
10306
10307        msg_info_prefix = "Start transcripts view to variants annotations"
10308
10309        log.debug(f"{msg_info_prefix}...")
10310
10311        # Default
10312        transcripts_table_default = "transcripts"
10313        transcripts_column_id_default = "transcript"
10314        transcripts_info_json_default = None
10315        transcripts_info_format_default = None
10316        transcripts_info_field_json_default = None
10317        transcripts_info_field_format_default = None
10318
10319        # Param
10320        if not param:
10321            param = self.get_param()
10322
10323        # Transcripts table
10324        if transcripts_table is None:
10325            transcripts_table = param.get("transcripts", {}).get(
10326                "table", transcripts_table_default
10327            )
10328
10329        # Transcripts column ID
10330        if transcripts_column_id is None:
10331            transcripts_column_id = param.get("transcripts", {}).get(
10332                "column_id", transcripts_column_id_default
10333            )
10334
10335        # Transcripts info json
10336        if transcripts_info_json is None:
10337            transcripts_info_json = param.get("transcripts", {}).get(
10338                "transcripts_info_json", transcripts_info_json_default
10339            )
10340
10341        # Transcripts info field JSON
10342        if transcripts_info_field_json is None:
10343            transcripts_info_field_json = param.get("transcripts", {}).get(
10344                "transcripts_info_field_json", transcripts_info_field_json_default
10345            )
10346        # if transcripts_info_field_json is not None and transcripts_info_json is None:
10347        #     transcripts_info_json = transcripts_info_field_json
10348
10349        # Transcripts info format
10350        if transcripts_info_format is None:
10351            transcripts_info_format = param.get("transcripts", {}).get(
10352                "transcripts_info_format", transcripts_info_format_default
10353            )
10354
10355        # Transcripts info field FORMAT
10356        if transcripts_info_field_format is None:
10357            transcripts_info_field_format = param.get("transcripts", {}).get(
10358                "transcripts_info_field_format", transcripts_info_field_format_default
10359            )
10360        # if (
10361        #     transcripts_info_field_format is not None
10362        #     and transcripts_info_format is None
10363        # ):
10364        #     transcripts_info_format = transcripts_info_field_format
10365
10366        # Variants table
10367        table_variants = self.get_table_variants()
10368
10369        # Check info columns param
10370        if (
10371            transcripts_info_json is None
10372            and transcripts_info_field_json is None
10373            and transcripts_info_format is None
10374            and transcripts_info_field_format is None
10375        ):
10376            return False
10377
10378        # Transcripts infos columns
10379        query_transcripts_infos_columns = f"""
10380            SELECT *
10381            FROM (
10382                DESCRIBE SELECT * FROM {transcripts_table}
10383                )
10384            WHERE "column_name" NOT IN ('#CHROM', 'POS', 'REF', 'ALT', '{transcripts_column_id}')
10385        """
10386        transcripts_infos_columns = list(
10387            self.get_query_to_df(query=query_transcripts_infos_columns)["column_name"]
10388        )
10389
10390        # View results
10391        clause_select = []
10392        clause_to_json = []
10393        clause_to_format = []
10394        for field in transcripts_infos_columns:
10395            clause_select.append(
10396                f""" regexp_split_to_table("{field}", ',') AS '{field}' """
10397            )
10398            clause_to_json.append(f""" '{field}': "{field}" """)
10399            clause_to_format.append(f""" "{field}" """)
10400
10401        # Update
10402        update_set_json = []
10403        update_set_format = []
10404
10405        # VCF header
10406        vcf_reader = self.get_header()
10407
10408        # Transcripts to info column in JSON
10409        if transcripts_info_json is not None:
10410
10411            # Create column on variants table
10412            self.add_column(
10413                table_name=table_variants,
10414                column_name=transcripts_info_json,
10415                column_type="JSON",
10416                default_value=None,
10417                drop=False,
10418            )
10419
10420            # Add header
10421            vcf_reader.infos[transcripts_info_json] = vcf.parser._Info(
10422                transcripts_info_json,
10423                ".",
10424                "String",
10425                "Transcripts in JSON format",
10426                "unknwon",
10427                "unknwon",
10428                self.code_type_map["String"],
10429            )
10430
10431            # Add to update
10432            update_set_json.append(
10433                f""" {transcripts_info_json}=t.{transcripts_info_json} """
10434            )
10435
10436        # Transcripts to info field in JSON
10437        if transcripts_info_field_json is not None:
10438
10439            log.debug(f"{msg_info_prefix} - Annotation in JSON format...")
10440
10441            # Add to update
10442            update_set_json.append(
10443                f""" 
10444                    INFO = concat(
10445                            CASE
10446                                WHEN INFO NOT IN ('', '.')
10447                                THEN INFO
10448                                ELSE ''
10449                            END,
10450                            CASE
10451                                WHEN CAST(t.{transcripts_info_json} AS VARCHAR) NOT IN ('', '.')
10452                                THEN concat(
10453                                    ';{transcripts_info_field_json}=',
10454                                    t.{transcripts_info_json}
10455                                )
10456                                ELSE ''
10457                            END
10458                            )
10459                """
10460            )
10461
10462            # Add header
10463            vcf_reader.infos[transcripts_info_field_json] = vcf.parser._Info(
10464                transcripts_info_field_json,
10465                ".",
10466                "String",
10467                "Transcripts in JSON format",
10468                "unknwon",
10469                "unknwon",
10470                self.code_type_map["String"],
10471            )
10472
10473        if update_set_json:
10474
10475            # Update query
10476            query_update = f"""
10477                UPDATE {table_variants}
10478                    SET {", ".join(update_set_json)}
10479                FROM
10480                (
10481                    SELECT
10482                        "#CHROM", POS, REF, ALT,
10483                            concat(
10484                            '{{',
10485                            string_agg(
10486                                '"' || "{transcripts_column_id}" || '":' ||
10487                                to_json(json_output)
10488                            ),
10489                            '}}'
10490                            )::JSON AS {transcripts_info_json}
10491                    FROM
10492                        (
10493                        SELECT
10494                            "#CHROM", POS, REF, ALT,
10495                            "{transcripts_column_id}",
10496                            to_json(
10497                                {{{",".join(clause_to_json)}}}
10498                            )::JSON AS json_output
10499                        FROM
10500                            (SELECT "#CHROM", POS, REF, ALT, "{transcripts_column_id}", {", ".join(clause_select)} FROM {transcripts_table})
10501                        WHERE "{transcripts_column_id}" IS NOT NULL
10502                        )
10503                    GROUP BY "#CHROM", POS, REF, ALT
10504                ) AS t
10505                WHERE {table_variants}."#CHROM" = t."#CHROM"
10506                    AND {table_variants}."POS" = t."POS"
10507                    AND {table_variants}."REF" = t."REF"
10508                    AND {table_variants}."ALT" = t."ALT"
10509            """
10510
10511            self.execute_query(query=query_update)
10512
10513        # Transcripts to info column in FORMAT
10514        if transcripts_info_format is not None:
10515
10516            # Create column on variants table
10517            self.add_column(
10518                table_name=table_variants,
10519                column_name=transcripts_info_format,
10520                column_type="VARCHAR",
10521                default_value=None,
10522                drop=False,
10523            )
10524
10525            # Add header
10526            vcf_reader.infos[transcripts_info_format] = vcf.parser._Info(
10527                transcripts_info_format,
10528                ".",
10529                "String",
10530                f"Transcripts annotations: 'transcript | {' | '.join(transcripts_infos_columns)}'",
10531                "unknwon",
10532                "unknwon",
10533                self.code_type_map["String"],
10534            )
10535
10536            # Add to update
10537            update_set_format.append(
10538                f""" {transcripts_info_format}=t.{transcripts_info_format} """
10539            )
10540
10541        # Transcripts to info field in JSON
10542        if transcripts_info_field_format is not None:
10543
10544            log.debug(f"{msg_info_prefix} - Annotation in structured format...")
10545
10546            # Add to update
10547            update_set_format.append(
10548                f""" 
10549                    INFO = concat(
10550                            CASE
10551                                WHEN INFO NOT IN ('', '.')
10552                                THEN INFO
10553                                ELSE ''
10554                            END,
10555                            CASE
10556                                WHEN CAST(t.{transcripts_info_format} AS VARCHAR) NOT IN ('', '.')
10557                                THEN concat(
10558                                    ';{transcripts_info_field_format}=',
10559                                    t.{transcripts_info_format}
10560                                )
10561                                ELSE ''
10562                            END
10563                            )
10564                """
10565            )
10566
10567            # Add header
10568            vcf_reader.infos[transcripts_info_field_format] = vcf.parser._Info(
10569                transcripts_info_field_format,
10570                ".",
10571                "String",
10572                f"Transcripts annotations: 'transcript | {' | '.join(transcripts_infos_columns)}'",
10573                "unknwon",
10574                "unknwon",
10575                self.code_type_map["String"],
10576            )
10577
10578        if update_set_format:
10579
10580            # Update query
10581            query_update = f"""
10582                UPDATE {table_variants}
10583                    SET {", ".join(update_set_format)}
10584                FROM
10585                (
10586                    SELECT
10587                        "#CHROM", POS, REF, ALT,
10588                            string_agg({transcripts_info_format}) AS {transcripts_info_format}
10589                    FROM 
10590                        (
10591                        SELECT
10592                            "#CHROM", POS, REF, ALT,
10593                            "{transcripts_column_id}",
10594                            concat(
10595                                "{transcripts_column_id}",
10596                                '|',
10597                                {", '|', ".join(clause_to_format)}
10598                            ) AS {transcripts_info_format}
10599                        FROM
10600                            (SELECT "#CHROM", POS, REF, ALT, "{transcripts_column_id}", {", ".join(clause_select)} FROM {transcripts_table})
10601                        )
10602                    GROUP BY "#CHROM", POS, REF, ALT
10603                ) AS t
10604                WHERE {table_variants}."#CHROM" = t."#CHROM"
10605                    AND {table_variants}."POS" = t."POS"
10606                    AND {table_variants}."REF" = t."REF"
10607                    AND {table_variants}."ALT" = t."ALT"
10608            """
10609
10610            self.execute_query(query=query_update)
10611
10612        return True

The transcript_view_to_variants function updates a variants table with information from transcripts in JSON format.

Parameters

transcripts_table: The transcripts_table parameter is used to specify the name of the table containing the transcripts data. If this parameter is not provided, the function will attempt to retrieve it from the param dictionary or use a default value of "transcripts"
transcripts_column_id: The transcripts_column_id parameter is used to specify the column in the transcripts_table that contains the unique identifier for each transcript. This identifier is used to match transcripts with variants in the database
transcripts_info_json: The transcripts_info_json parameter is used to specify the name of the column in the variants table where the transcripts information will be stored in JSON format. This parameter allows you to define the column in the variants table that will hold the JSON-formatted information about transcripts
transcripts_info_field_json: The transcripts_info_field_json parameter is used to specify the field in the VCF header that will contain information about transcripts in JSON format. This field will be added to the VCF header as an INFO field with the specified name
transcripts_info_format: The transcripts_info_format parameter is used to specify the format of the information about transcripts that will be stored in the variants table. This format can be used to define how the transcript information will be structured or displayed within the variants table
transcripts_info_field_format: The transcripts_info_field_format parameter is used to specify the field in the VCF header that will contain information about transcripts in a specific format. This field will be added to the VCF header as an INFO field with the specified name
param: The param parameter in the transcript_view_to_variants method is a dictionary that contains various configuration settings related to transcripts. It is used to provide default values for certain parameters if they are not explicitly provided when calling the method. The param dictionary can be passed as an argument

Returns

The function transcript_view_to_variants returns a boolean value. It returns True if the operation is successful and False if certain conditions are not met.